returnn.frontend.init

Common parameter initialization functions.

https://github.com/rwth-i6/returnn/wiki/Parameter-initialization

class returnn.frontend.init.ParamInit

API for param init

class returnn.frontend.init.Normal(stddev: float, *, truncated: bool = True, dtype: str | None = None)

Initialization by normal distribution (truncated by default), independent of the dimensions (fan in/out).

See VarianceScaling and derivatives for variants which depend on fan in/out.

class returnn.frontend.init.VarianceScaling(scale: float | None = None, mode: str | None = None, distribution: str | None = None, dtype: str | None = None)

Provides a generalized way for initializing weights. All the common initialization methods are special cases such as Xavier Glorot and Kaiming He.

Code adopted from TensorFlow VarianceScaling.

scale = 1.0

mode = 'fan_in'

distribution = 'truncated_normal'

dtype: str

class returnn.frontend.init.Glorot(scale: float | None = None, mode: str | None = None, distribution: str | None = None, dtype: str | None = None)

Xavier Glorot (http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf). scale 1, fan_avg, uniform

scale = 1.0

mode = 'fan_avg'

distribution = 'uniform'

class returnn.frontend.init.He(scale: float | None = None, mode: str | None = None, distribution: str | None = None, dtype: str | None = None)

Kaiming He (https://arxiv.org/pdf/1502.01852.pdf). scale 2, fan_in, normal

scale = 2.0

mode = 'fan_in'

distribution = 'normal'

returnn.frontend.init.HeNormal

alias of He

class returnn.frontend.init.HeUniform(scale: float | None = None, mode: str | None = None, distribution: str | None = None, dtype: str | None = None)

He-init (He) but using a uniform distribution. scale 2, fan_in, uniform

distribution = 'uniform'