`returnn.frontend.rand`¶

Random number generator utilities.

Note on the global seed:

In the TF engine, we have this in _init_network:

tf_random_seed = 42
net_random_seed = epoch
if self.config.opt_typed_value("random_seed", None):
    seed = self.config.int("random_seed", None)
    net_random_seed = (epoch * 3 + seed * 5 + 7) % (2**31)
    tf_random_seed = (net_random_seed * 2 + 3) % (2**31)
tf_compat.v1.set_random_seed(tf_random_seed)

_init_network is only called in the beginning and then only when needed (e.g. different model due to pretrain).

The net_random_seed is used inside the TFNetwork for:

self.random = numpy.random.RandomState(rnd_seed)

This in turns is used for param init in the TF layers, like:

fwd_weights_initializer = get_initializer(
    forward_weights_init, seed=self.network.random.randint(2**31), eval_local_ns={"layer": self}
)

It is also used by the RandomLayer when static=True:

# static is True
if seed is None:
    seed = self.network.random.randint(2**31, size=[32], dtype="uint32")

So, it means, with static=True, the random numbers are always the same in each execution step. To reflect that in an eager-based backend, we need to reset the static-random-state in the beginning of each step.

returnn.frontend.rand.set_random_seed(seed: int)[source]¶

This initializes the random state of the backend and also the step-based random state (see get_static_step_based_seed(), only used when static=True in random()).

Call this at the beginning of the program (after the RF backend was selected), or when the model and computation graph is supposed to be reinitialized or at the beginning of each epoch.

Parameters:: seed – should depend on epoch or step

returnn.frontend.rand.get_random_state() → Dict[str, bytes][source]¶

Returns:: current random state, for serialization, to be able to restore it later

returnn.frontend.rand.set_random_state(state: Dict[str, bytes])[source]¶

Recovers the random state.

There are many potential cases where we cannot recover the state (e.g. different backend version, different hardware, …), In this case, a run without interruption is not the same as a run with interruption.

We still assume that set_random_seed() was called before in any case.

Parameters:: state – as returned by get_random_state()

returnn.frontend.rand.reset_step_random_state()[source]¶: When static=True is used in random(), the random state is reset to the beginning of the step. So this should be called in the beginning of each step. Also see the module docstring.

returnn.frontend.rand.get_static_step_based_seed(*, size=None) → int | ndarray[source]¶

This is intended as a static seed for random() when static=True is used.

Returns:: from the static step-based random state, get a seed

Generates random numbers from uniform or normal or truncated normal distribution.

There will be no gradients to mean, stddev, bound, minval, maxval!

In case of TensorFlow: This uses the TensorFlow stateless random ops internally, i.e. all the state handling is explicit. The state var can be explicitly provided and initialized via RandomStateInitLayer, or when not provided it will be automatically created.

There are two possible distinct use cases:

For any randomness in the model, e.g. dropout. So each session.run step will produce a new random number and advance the random state.
To initialize parameters via the config, using VariableLayer with the init_by_layer option. This will only be called once when initializing the parameters. For this use case, we do not want to keep a random state var. You can just pass static=False. Alternatively you could also pass the output of a RandomStateInitLayer as state.

Parameters:

dims
dtype
device
sparse_dim
feature_dim
distribution (str) – “uniform”, “normal” or “truncated_normal”
mean (int|float|Tensor|None)
stddev (int|float|Tensor|None)
bound (int|float|Tensor|None) – for uniform, defining the range [-bound, bound)
minval (int|float|Tensor|None) – for uniform, inclusive
maxval (int|float|Tensor|None) – for uniform, exclusive
seed (int|list[int]|numpy.ndarray|None) – Only for the case static=True. If not given, uses self.network.random.randint, i.e. then it is controlled by the global seed setting, and every layer would get its own seed. If you specify it explicitly, make sure every RandomLayer uses a different seed, otherwise you would get the same random numbers everywhere.
algorithm (str|tf.random.Algorithm|None) – see RandomStateInitLayer
explicit_state (Tensor|None) – You can pass the state explicitly here. If not given, will be created automatically, and updated automatically. You could pass a VariableLayer with initial value via RandomStateInitLayer, or directly a RandomStateInitLayer. If auto_update_state is True, it must be a variable, and every time a new random number is created, this variable is updated. Otherwise (default), it will not be updated automatically.
auto_update_state (bool|None) – only used when you pass an explicit state
static (bool|None) – if no state at all should be used. it just relies on the seed then.
out – if given, will directly write into it, if possible by backend

Returns:

random values

returnn.frontend.rand.random_uniform(dims: Sequence[Dim], *, dtype: str | None = None, device: str | None = None, sparse_dim: Dim | None = None, feature_dim: Dim | None = None, minval: int | float | Tensor = 0, maxval: int | float | Tensor = 1, seed: int | Sequence[int] | ndarray | None = None, algorithm: str | None = None, explicit_state: Tensor | None = None, auto_update_state: bool | None = None, static: bool | None = None, out: Tensor | None = None)[source]¶: See random(). random() with distribution="uniform". maxval is exclusive.

returnn.frontend.rand.random_normal(dims: Sequence[Dim], *, dtype: str | None = None, device: str | None = None, sparse_dim: Dim | None = None, feature_dim: Dim | None = None, mean: int | float | Tensor | None = 0.0, stddev: int | float | Tensor | None = 1.0, seed: int | Sequence[int] | ndarray | None = None, algorithm: str | None = None, explicit_state: Tensor | None = None, auto_update_state: bool | None = None, static: bool | None = None, out: Tensor | None = None)[source]¶: See random(). random() with distribution="normal".

returnn.frontend.rand.random_truncated_normal(dims: Sequence[Dim], *, dtype: str | None = None, device: str | None = None, sparse_dim: Dim | None = None, feature_dim: Dim | None = None, mean: int | float | Tensor | None = 0.0, stddev: int | float | Tensor | None = 1.0, minval: int | float | Tensor | None = None, maxval: int | float | Tensor | None = None, seed: int | Sequence[int] | ndarray | None = None, algorithm: str | None = None, explicit_state: Tensor | None = None, auto_update_state: bool | None = None, static: bool | None = None, out: Tensor | None = None)[source]¶: See random(). random() with distribution="truncated_normal".

returnn.frontend.rand.random_choice_without_replacement(*, log_probs: Tensor, axis: Dim | Sequence[Dim], num_samples_dim: Dim, noise_scale: float | Tensor = 1.0) → Tensor | Sequence[Tensor][source]¶

Randomly sample without replacement.

Parameters:

log_probs – {log_probs_dims…, axis}
axis – same as in top_k()
num_samples_dim – how many samples to draw
noise_scale – scale the noise. with scale=0, you get top_k().

Returns:

random indices shape {log_probs_dims…, num_samples_dim} -> axis. if axis was a sequence, will return a sequence of tensors.

returnn.frontend.rand.random_choice_with_replacement(dims: Sequence[Dim], *, probs: Tensor, axis: Dim) → Tensor[source]¶

Randomly sample with replacement.

Parameters:

dims – {common_dims…, new_dims…}. Defines how many samples to draw. Defines the output shape.
probs – {common_dims…, axis}
axis – feature axis, where to sample from

Returns:

random indices shape dims -> axis.

returnn.frontend.rand¶

`returnn.frontend.rand`¶