ukiBjUdZddlmZddlmZddlZddlZddlmZmZm Z m Z ddl Z ddl mZddl mZddl m Zdd l mZdd lmZdd lmZdd lmZdd lmZmZmZddlmZedZeZeZ eZ!eZ"eezdzZ#de$d<eejJGdde Z&e d& d'dZ'e d& d'dZ(e d( d)dZ)e d* d+dZ*e d* d,dZ+e d- d.dZ,e d/ d0dZ- d1dZ. d2dZ/e d3 d4dZ0e d3 d5dZ1e1Z2e d3 d5dZ3e3Z4e d3 d5d Z5e d3 d5d!Z6e d3 d5d"Z7e7Z8e d3 d5d#Z9e9Z:e d6 d7d$Z;e d6 d7d%Z r!rcb|tjn|}tj|||S)a An initializer that returns a constant array full of zeros. The ``key`` argument is ignored. >>> import jax, jax.numpy as jnp >>> jax.nn.initializers.zeros(jax.random.key(42), (2, 3), jnp.float32) Array([[0., 0., 0.], [0., 0., 0.]], dtype=float32) r)r default_float_dtypejnpzerosrrrrs rr/r/<s++0-& $ $ &U% 5%l ;;r!cb|tjn|}tj|||S)aAn initializer that returns a constant array full of ones. The ``key`` argument is ignored. >>> import jax, jax.numpy as jnp >>> jax.nn.initializers.ones(jax.random.key(42), (3, 2), jnp.float32) Array([[1., 1.], [1., 1.], [1., 1.]], dtype=float32) r,)r r-r.onesr0s rr2r2Ms++0-& $ $ &U% %\ ::r!c(|df dfd }|S)aBuilds an initializer that returns arrays full of a constant ``value``. Args: value: the constant value with which to fill the initializer. dtype: optional; the initializer's default dtype. >>> import jax, jax.numpy as jnp >>> initializer = jax.nn.initializers.constant(-7) >>> initializer(jax.random.key(42), (2, 3), jnp.float32) Array([[-7., -7., -7.], [-7., -7., -7.]], dtype=float32) Nc~|tjn|}t|d}tj|||S)Nznn.initializers.constant)rdevice)r r-rr.full)rrrrvalues rinitzconstant..initns;-2MF & & (uE(7QRL 88E5l CCr!r#r*)r7rr8s` rconstantr9_s@"-2+/DD)D)D5:D +r!c(|df dfd }|S)aYBuilds an initializer that returns real uniformly-distributed random arrays. Args: scale: optional; the upper bound of the random distribution. dtype: optional; the initializer's default dtype. Returns: An initializer that returns arrays whose values are uniformly distributed in the range ``[0, scale)``. >>> import jax, jax.numpy as jnp >>> initializer = jax.nn.initializers.uniform(10.0) >>> initializer(jax.random.key(42), (2, 3), jnp.float32) # doctest: +SKIP Array([[7.298188 , 8.691938 , 8.7230015], [2.0818567, 1.8662417, 5.5022564]], dtype=float32) Nc|tjn|}tj||||t j |zSNr,)r r-r uniformr.array)rrrrscales rr8zuniform..initsH-2MF & & (uE >>#ue'3 57:yy7N OOr!r#r*)r?rr8s` rr=r=w@*-2+/OO)O)O5:O +r!c(|df dfd }|S)axBuilds an initializer that returns real normally-distributed random arrays. Args: stddev: optional; the standard deviation of the distribution. dtype: optional; the initializer's default dtype. Returns: An initializer that returns arrays whose values are normally distributed with mean ``0`` and standard deviation ``stddev``. >>> import jax, jax.numpy as jnp >>> initializer = jax.nn.initializers.normal(5.0) >>> initializer(jax.random.key(42), (2, 3), jnp.float32) # doctest: +SKIP Array([[ 3.0613258 , 5.6129413 , 5.6866574 ], [-4.063663 , -4.4520254 , 0.63115686]], dtype=float32) Nc|tjn|}tj||||t j |zSr<)r r-r normalr.r>)rrrrstddevs rr8znormal..initsH-2MF & & (uE ==eU&2 469ii6N OOr!r#r*)rDrr8s` rrCrCr@r!c0|df dfd }|S)aBuilds an initializer that returns truncated-normal random arrays. Args: stddev: optional; the standard deviation of the untruncated distribution. Note that this function does not apply the stddev correction as is done in the variancescaling initializers, and users are expected to apply this correction themselves via the stddev arg if they wish to employ it. dtype: optional; the initializer's default dtype. lower: Float representing the lower bound for truncation. Applied before the output is multiplied by the stddev. upper: Float representing the upper bound for truncation. Applied before the output is multiplied by the stddev. Returns: An initializer that returns arrays whose values follow the truncated normal distribution with mean ``0`` and standard deviation ``stddev``, and range :math:`\rm{lower * stddev} < x < \rm{upper * stddev}`. >>> import jax, jax.numpy as jnp >>> initializer = jax.nn.initializers.truncated_normal(5.0) >>> initializer(jax.random.key(42), (2, 3), jnp.float32) # doctest: +SKIP Array([[ 2.9047365, 5.2338114, 5.29852 ], [-3.836303 , -4.192359 , 0.6022964]], dtype=float32) Nc|tjn|}tj||||t j |zSr<)r r-r truncated_normalr.r>)rrrrlowerrDuppers rr8ztruncated_normal..initsM-2MF & & (uE  " " UE5%! #%(YYvu%= >>r!r#r*)rDrrHrIr8s` `` rrGrGs;>-2+/>>)>)>5:> +r!cVt|tr+|dk(r&t|dkrtdt|dt|tr||}n&t j |Dcgc]}|| c}}t|tr||}n&t j |Dcgc]}|| c}}t|tr||}n&t j |Dcgc]}|| c}}t j ||z |z |z }||z} ||z} | | fScc}wcc}wcc}w)z Compute effective input and output sizes for a linear or convolutional layer. Axes not in in_axis, out_axis, or batch_axis are assumed to constitute the "receptive field" of a convolution (kernel spatial dimensions). z*Can't compute input and output sizes of a zd-dimensional weights tensor with default in_axis. Must be at least 2D or specify in_axis explicitly.) isinstanceintlen ValueErrormathprod) rin_axisout_axis batch_axisin_sizeiout_size batch_sizereceptive_field_sizefan_infan_outs r _compute_fansr]s!'R-CJ!O  4SZLA   GnGii73aq34G#XHyyH5q%(56H C z"Jj9E!H9:J5)G3h>K ) )& + +' 46:s% D! D! D&ctj|\}}tjd|jj }t j|}tjdtj|||zj|}dtjztj|||j|z}|tjd|zzS)zj Sample uniform random values within a disk on the complex plane, with zero mean and unit variance. r?)r splitnpr>realrr to_complex_dtyper.sqrtr=astypepiexp)rrrkey_r key_theta real_dtyperthetas r_complex_uniformrns\\#&%xx5!&&,,*  ! !* -% hhq6>>% ;;<CCEJ! bee)fnnYzBII%P P% SWWR%Z  r!c`tj|\}}tjd|jj }t j|}dtjtj|dz |z tj|||j|z}tjtjd|z  }dtjztj|||j|z} |tjd| zzS)z Sample random values from a centered normal distribution on the complex plane, whose modulus is truncated to `upper`, and the variance before the truncation is one. rrLr_r`)r rarbr>rcrr rdr.rhr=rfrelogrg) rrIrrrirjrktrlrms r_complex_truncated_normalrrs\\#&%xx5!&&,,*  ! !* -% CGGCII mU3 44 uj 1 8 8 ?@! hhA! bee)fnnYzBII%P P% SWWR%Z  r!c<|df dfd }|S)a# Initializer that adapts its scale to the shape of the weights tensor. With ``distribution="truncated_normal"`` or ``distribution="normal"``, samples are drawn from a (truncated) normal distribution with a mean of zero and a standard deviation (after truncation, if applicable) of :math:`\sqrt{\frac{scale}{n}}`, where `n` is, for each ``mode``: * ``"fan_in"``: the number of inputs * ``"fan_out"``: the number of outputs * ``"fan_avg"``: the arithmetic average of the numbers of inputs and outputs * ``"fan_geo_avg"``: the geometric average of the numbers of inputs and outputs This initializer can be configured with ``in_axis``, ``out_axis``, and ``batch_axis`` to work with general convolutional or dense layers; axes that are not in any of those arguments are assumed to be the "receptive field" (convolution kernel spatial axes). With ``distribution="truncated_normal"``, the absolute values of the samples are truncated at 2 standard deviations before scaling. With ``distribution="uniform"``, samples are drawn from: * a uniform interval, if `dtype` is real, or * a uniform disk, if `dtype` is complex, with a mean of zero and a standard deviation of :math:`\sqrt{\frac{scale}{n}}` where `n` is defined above. Args: scale: scaling factor (positive float). mode: one of ``"fan_in"``, ``"fan_out"``, ``"fan_avg"``, and ``"fan_geo_avg"``. distribution: random distribution to use. One of ``"truncated_normal"``, ``"normal"`` and ``"uniform"``. in_axis: axis or sequence of axes of the input dimension in the weights array. out_axis: axis or sequence of axes of the output dimension in the weights array. batch_axis: axis or sequence of axes in the weight array that should be ignored. dtype: the dtype of the weights. Nc tj|}|tjn|}t | \}} dk(r|}n2 dk(r|}n* dk(r ||zdz }n dk(r ||zdz}nt d t j|z |} d k(rtj|tjrJt j|t jd |z }tj|d d||| |zSt j|t jd |z }t|d|||zS dk(r/tj|||| t j|zS dk(rztj|tjr3tj |||d| t jd|zzSt#|||t j|zSt d )Nr[r\fan_avgr_ fan_geo_avgg?z/invalid mode for variance scaling initializer: rrGg۶%?rKr,gVr?rCr=z7invalid distribution for variance scaling initializer: )r canonicalize_shaper r-r]rPr.r> issubdtyperbfloatingrer rGrrrCr=rn)rrrrr[r\ denominatorvariancerDrU distributionrSmoderTr?s rr8zvariance_scaling..initNs  # #E *E,1MF & & (uE#E7HjIOFG xv  'K  6G+;q*@K  v/?C.G  9$@ BByy,E:H))   5"++ .(#cii0BE&JJ&&sB5%4@BDJK K(#cii0BE&JJ(a>GG  ! ]]3u(468;8JKK  "   5"++ .~~c5%+79;>88AL;QR R UE2SXXh5GGG PQ]P^_ ``r!r#r*)r?rrrSrTrUrr8s`````` rvariance_scalingrsHn-2+/$a$a)$a)$a5:$a$aL +r!c &tddd||||S)aBuilds a Glorot uniform initializer (aka Xavier uniform initializer). A `Glorot uniform initializer`_ is a specialization of :func:`jax.nn.initializers.variance_scaling` where ``scale = 1.0``, ``mode="fan_avg"``, and ``distribution="uniform"``. Args: in_axis: axis or sequence of axes of the input dimension in the weights array. out_axis: axis or sequence of axes of the output dimension in the weights array. batch_axis: axis or sequence of axes in the weight array that should be ignored. dtype: the dtype of the weights. Returns: An initializer. Examples: >>> import jax, jax.numpy as jnp >>> initializer = jax.nn.initializers.glorot_uniform() >>> initializer(jax.random.key(42), (2, 3), jnp.float32) # doctest: +SKIP Array([[ 0.50350785, 0.8088631 , 0.81566876], [-0.6393332 , -0.6865721 , 0.11003882]], dtype=float32) .. _Glorot uniform initializer: http://proceedings.mlr.press/v9/glorot10a.html ?rur=rSrTrUrrrs rglorot_uniformrvs#B #y)W#+ % QQr!c &tddd||||S)aBuilds a Glorot normal initializer (aka Xavier normal initializer). A `Glorot normal initializer`_ is a specialization of :func:`jax.nn.initializers.variance_scaling` where ``scale = 1.0``, ``mode="fan_avg"``, and ``distribution="truncated_normal"``. Args: in_axis: axis or sequence of axes of the input dimension in the weights array. out_axis: axis or sequence of axes of the output dimension in the weights array. batch_axis: axis or sequence of axes in the weight array that should be ignored. dtype: the dtype of the weights. Returns: An initializer. Examples: >>> import jax, jax.numpy as jnp >>> initializer = jax.nn.initializers.glorot_normal() >>> initializer(jax.random.key(42), (2, 3), jnp.float32) # doctest: +SKIP Array([[ 0.41770416, 0.75262755, 0.7619329 ], [-0.5516644 , -0.6028657 , 0.08661086]], dtype=float32) .. _Glorot normal initializer: http://proceedings.mlr.press/v9/glorot10a.html rrurGrrrs r glorot_normalrs$B #y*>> import jax, jax.numpy as jnp >>> initializer = jax.nn.initializers.lecun_uniform() >>> initializer(jax.random.key(42), (2, 3), jnp.float32) # doctest: +SKIP Array([[ 0.56293887, 0.90433645, 0.9119454 ], [-0.71479625, -0.7676109 , 0.12302713]], dtype=float32) .. _Lecun uniform initializer: https://arxiv.org/abs/1706.02515 rr[r=rrrs r lecun_uniformr#B #xG#+ % QQr!c &tddd||||S)aBuilds a Lecun normal initializer. A `Lecun normal initializer`_ is a specialization of :func:`jax.nn.initializers.variance_scaling` where ``scale = 1.0``, ``mode="fan_in"``, and ``distribution="truncated_normal"``. Args: in_axis: axis or sequence of axes of the input dimension in the weights array. out_axis: axis or sequence of axes of the output dimension in the weights array. batch_axis: axis or sequence of axes in the weight array that should be ignored. dtype: the dtype of the weights. Returns: An initializer. Examples: >>> import jax, jax.numpy as jnp >>> initializer = jax.nn.initializers.lecun_normal() >>> initializer(jax.random.key(42), (2, 3), jnp.float32) # doctest: +SKIP Array([[ 0.46700746, 0.8414632 , 0.8518669 ], [-0.61677957, -0.67402434, 0.09683388]], dtype=float32) .. _Lecun normal initializer: https://arxiv.org/abs/1706.02515 rr[rGrrrs r lecun_normalr$B #x);W#+ % QQr!c &tddd||||S)aBuilds a He uniform initializer (aka Kaiming uniform initializer). A `He uniform initializer`_ is a specialization of :func:`jax.nn.initializers.variance_scaling` where ``scale = 2.0``, ``mode="fan_in"``, and ``distribution="uniform"``. Args: in_axis: axis or sequence of axes of the input dimension in the weights array. out_axis: axis or sequence of axes of the output dimension in the weights array. batch_axis: axis or sequence of axes in the weight array that should be ignored. dtype: the dtype of the weights. Returns: An initializer. Examples: >>> import jax, jax.numpy as jnp >>> initializer = jax.nn.initializers.he_uniform() >>> initializer(jax.random.key(42), (2, 3), jnp.float32) # doctest: +SKIP Array([[ 0.79611576, 1.2789248 , 1.2896855 ], [-1.0108745 , -1.0855657 , 0.17398663]], dtype=float32) .. _He uniform initializer: https://arxiv.org/abs/1502.01852 @r[r=rrrs r he_uniformr rr!c &tddd||||S)aBuilds a He normal initializer (aka Kaiming normal initializer). A `He normal initializer`_ is a specialization of :func:`jax.nn.initializers.variance_scaling` where ``scale = 2.0``, ``mode="fan_in"``, and ``distribution="truncated_normal"``. Args: in_axis: axis or sequence of axes of the input dimension in the weights array. out_axis: axis or sequence of axes of the output dimension in the weights array. batch_axis: axis or sequence of axes in the weight array that should be ignored. dtype: the dtype of the weights. Returns: An initializer. Examples: >>> import jax, jax.numpy as jnp >>> initializer = jax.nn.initializers.he_normal() >>> initializer(jax.random.key(42), (2, 3), jnp.float32) # doctest: +SKIP Array([[ 0.6604483 , 1.1900088 , 1.2047218 ], [-0.87225807, -0.95321447, 0.1369438 ]], dtype=float32) .. _He normal initializer: https://arxiv.org/abs/1502.01852 rr[rGrrrs r he_normalr0rr!c,|df dfd }|S)a Builds an initializer that returns uniformly distributed orthogonal matrices. If the shape is not square, the matrices will have orthonormal rows or columns depending on which side is smaller. Args: scale: the upper bound of the uniform distribution. column_axis: the axis that contains the columns that should be orthogonal. dtype: the default dtype of the weights. Returns: An orthogonal initializer. Examples: >>> import jax, jax.numpy as jnp >>> initializer = jax.nn.initializers.orthogonal() >>> initializer(jax.random.key(42), (2, 3), jnp.float32) # doctest: +SKIP Array([[ 3.9026976e-01, 7.2495741e-01, -5.6756169e-01], [ 8.8047469e-01, -4.7409311e-01, -1.3157725e-04]], dtype=float32) Nc |t|tjn|}t|dkr t dt j ||z|}}tj||d||}tj|ttj||fz}tj|d}tj||zS)Nr_z3orthogonal initializer requires at least a 2D shaper*rx)rr r-rOrPrQrRr orthogonalr.reshapetuplerbdeletemoveaxisr>) rrrrn_rowsn_colsQ column_axisr?s rr8zorthogonal..initps ,1MF & & (uE 5zA~ L MMYYu%{);;U;=OFF#vr5&9A AuRYYuk:;u[?Q>SSTA QK(A 99UE "Q &&r!r#r*r?rrr8s`` rrrVs;8-2+/ ' ') ') '5: ' +r!c,|df dfd }|S)ae Builds an initializer for delta orthogonal kernels. Args: scale: the upper bound of the uniform distribution. column_axis: the axis that contains the columns that should be orthogonal. dtype: the default dtype of the weights. Returns: A `delta orthogonal initializer`_. The shape passed to the initializer must be 3D, 4D, or 5D. Examples: >>> import jax, jax.numpy as jnp >>> initializer = jax.nn.initializers.delta_orthogonal() >>> initializer(jax.random.key(42), (3, 3, 3), jnp.float32) # doctest: +SKIP Array([[[ 0. , 0. , 0. ], [ 0. , 0. , 0. ], [ 0. , 0. , 0. ]], [[ 0.27858758, -0.7949833 , -0.53887904], [ 0.9120717 , 0.04322892, 0.40774566], [-0.30085585, -0.6050892 , 0.73712474]], [[ 0. , 0. , 0. ], [ 0. , 0. , 0. ], [ 0. , 0. , 0. ]]], dtype=float32) .. _delta orthogonal initializer: https://arxiv.org/abs/1806.05393 Nct|t|tjn|}t|dvr t d|d|dkr t dt |}|||dd}t j||}t|dk(r+|d }|j|d z d zd fj|St|d k(r5|dd \}} |j|d z d z| d z d zd fj|S|dd\}} } |j|d z d z| d z d z| d z d zd fj|S)N)ryz;Delta orthogonal initializer requires a 3D, 4D or 5D shape.rxrKz/`fan_in` must be less or equal than `fan_out`. )r?rrrwryrrLr_.r) rr r-rOrPrr.r/atset) rrrr ortho_init ortho_matrixWkk1k2k3rr?s rr8zdelta_orthogonal..inits` ,1MF & & (uE 5z"   !! Ry59 H II%[NJc5:.L %u%A 5zQ (a TT1Q3(C- $ $\ 22 UqRayfb" TT2a4!)bdQY+ , 0 0 >>!9jb"b TT2a4!)bdQYA 36 7 ; ;L IIr!r#r*rs`` rdelta_orthogonalrsAN-2+/JJ)J)J5:J0 +r!r"r#r)r7rrr$r%r){Gz?N)r? RealNumericrr$r%r)rDrrr$r%r)rNgr) rDrrr$rHrrIrr%r)rKrxr*) r Sequence[int]rSint | Sequence[int]rTrrUrr%ztuple[float, float])rrrrrrr%r) rrrIrrrrrr%r)rKrxr*N)r?rrzTLiteral['fan_in'] | Literal['fan_out'] | Literal['fan_avg'] | Literal['fan_geo_avg']rzDLiteral['truncated_normal'] | Literal['normal'] | Literal['uniform']rSrrTrrUrrr$r%r) rSrrTrrUrrr$r%r)rrxN)r?rrrNrr$r%r)=r) __future__rcollections.abcrrQtypingrrrrr rbjax._srcr r r.r jax._src.named_shardingr jax._src.partition_specrjax._src.sharding_implsrjax._src.typingrrr jax._src.utilrexportDTypeLikeFloatDTypeLikeComplexDTypeLikeInexactrr__annotations__runtime_checkablerr/r2r9r=rCrGr]rnrrrrxavier_uniformr xavier_normalrrrkaiming_uniformrkaiming_normalrrr*r!rrs% #$ 44!11933$ ) * *]:TAA(,0*.<<(<(<49<< +/)-;;';';38;;".2+7B.!%-1*6A6!%,0)5@6+/6:*.*-$3$'$($3>$$L132446!.!/!2!+ !!F !) !! !&+ !!%2!%*!/4!" "$"$$&#'Z Z\Z)Z  Z Z"Z !ZZZx24355748!Q0!Q2!Q2!Q>I!Q!QF 13244637!Q/!Q1!Q1!Q=H!Q!QF 13244637!Q/!Q1!Q1!Q=H!Q!QF02133526!Q.!Q0!Q0!Q