Generative Layers

Positional Encoding Layer

class returnn.tf.layers.rec.PositionalEncodingLayer(axis=<class 'returnn.util.basic.NotSpecified'>, add_to_input=False, constant=-1, offset=None, **kwargs)

Provides positional encoding in the form of (batch, time, n_out) or (time, batch, n_out) where n_out is the number of channels, if it is run outside a RecLayer, and (batch, n_out) or (n_out, batch) if run inside a RecLayer, where it will depend on the current time frame.

Assumes one source input with a time dimension if outside a RecLayer. With add_to_input, it will calculate x + input, and the output shape is the same as the input

The positional encoding is the same as in Tensor2Tensor. See returnn.tf.util.basic.get_positional_encoding().

Parameters:

• axis (Dim|str|NotSpecified) – if not specified, check for time_dim_axis, otherwise assume rec step

• add_to_input (bool) – will add the signal to the input

• constant (int) – if positive, always output the corresponding positional encoding.

• offset (None|LayerBase) – Specify the offset to be added to positions. Expect shape (batch, time) or (batch,).

layer_class: Optional[str] = 'positional_encoding'

recurrent = True

classmethod transform_config_dict(d, network, get_layer)

Parameters:

• d (dict[str])

• network (returnn.tf.network.TFNetwork)

• get_layer (((str)->LayerBase))

classmethod get_out_data_from_opts(name, network, add_to_input=False, sources=(), **kwargs)

Parameters:

• name (str)

• network (returnn.tf.network.TFNetwork)

• add_to_input (bool)

• sources (list[LayerBase])

Return type:

Data

classmethod get_rec_initial_extra_outputs(batch_dim, rec_layer, network, **kwargs)

Parameters:

• batch_dim (tf.Tensor) – for this layer, might be with beam

• rec_layer (returnn.tf.layers.rec.RecLayer|LayerBase|None) – for the scope

• network (returnn.tf.network.TFNetwork)

Return type:

dict[str,tf.Tensor]

classmethod get_rec_initial_extra_outputs_shape_invariants(rec_layer, network, **kwargs)

Parameters:

• rec_layer (returnn.tf.layers.rec.RecLayer|LayerBase|None) – for the scope

• network (returnn.tf.network.TFNetwork)

Returns:

optional shapes for the tensors by get_rec_initial_extra_outputs

Return type:

dict[str,tf.TensorShape]

input_data: Optional[Data]

kwargs: Optional[Dict[str]]

output_before_activation: Optional[OutputWithActivation]

output_loss: Optional[tf.Tensor]

rec_vars_outputs: Dict[str, tf.Tensor]

search_choices: Optional[SearchChoices]

params: Dict[str, tf.Variable]

saveable_param_replace: Dict[tf.Variable, Union['tensorflow.python.training.saver.BaseSaverBuilder.SaveableObject', None]]

stats: Dict[str, tf.Tensor]

Range Layer

class returnn.tf.layers.basic.RangeLayer(limit, start=0, delta=1, dtype=None, sparse=False, out_spatial_dim=None, **kwargs)

Generic wrapper around tf.range. See also RangeInAxisLayer.

Parameters:

• limit (int|float)

• start (int|float)

• delta (int|float)

• dtype (str|None)

• sparse (bool)

• out_spatial_dim (Dim|None)

layer_class: Optional[str] = 'range'

classmethod transform_config_dict(d, network, get_layer)

Parameters:

• d (dict[str])

• network (returnn.tf.network.TFNetwork)

• get_layer ((str)->LayerBase)

classmethod get_out_data_from_opts(name, limit, start=0, delta=1, dtype=None, sparse=False, out_spatial_dim=None, **kwargs)

Parameters:

• name (str)

• limit (int|float)

• start (int|float)

• delta (int|float)

• dtype (str|None)

• sparse (bool)

• out_spatial_dim (Dim|None)

Return type:

Data

kwargs: Optional[Dict[str]]

output_before_activation: Optional[OutputWithActivation]

output_loss: Optional[tf.Tensor]

rec_vars_outputs: Dict[str, tf.Tensor]

search_choices: Optional[SearchChoices]

params: Dict[str, tf.Variable]

saveable_param_replace: Dict[tf.Variable, Union['tensorflow.python.training.saver.BaseSaverBuilder.SaveableObject', None]]

stats: Dict[str, tf.Tensor]

Range in Axis Layer

class returnn.tf.layers.basic.RangeInAxisLayer(axis, dtype='int32', unbroadcast=False, keepdims=False, sparse=False, **kwargs)

Assume that the input is e.g. (B,T,D), and you specify axis=”T”, you will get (T,), where the specified axis is filled with tf.range. See also RangeLayer.

Parameters:

• axis (str|Dim)

• dtype (str)

• unbroadcast (bool) – DEPRECATED, unsupported, and not needed

• keepdims (bool) – DEPRECATED, unsupported, and not needed

• sparse (bool)

layer_class: Optional[str] = 'range_in_axis'

recurrent = True

classmethod get_out_data_from_opts(name, sources, axis, dtype='int32', sparse=False, **kwargs)

Parameters:

• name (str)

• sources (list[LayerBase])

• axis (str|Dim)

• dtype (str)

• sparse (bool)

kwargs: Optional[Dict[str]]

output_before_activation: Optional[OutputWithActivation]

output_loss: Optional[tf.Tensor]

rec_vars_outputs: Dict[str, tf.Tensor]

search_choices: Optional[SearchChoices]

params: Dict[str, tf.Variable]

saveable_param_replace: Dict[tf.Variable, Union['tensorflow.python.training.saver.BaseSaverBuilder.SaveableObject', None]]

stats: Dict[str, tf.Tensor]

Range from Length Layer

class returnn.tf.layers.basic.RangeFromLengthLayer(dtype='int32', sparse=False, out_spatial_dim=None, **kwargs)

Given some dynamic sequence lengths as input, this creates a tf.range over the implied dimension. As a side effect, this can create a new dyn dim tag for the given sequence lengths. This side effect can be the main functionality in certain use cases. See also RangeInAxisLayer.

Consider the example:

y: {class: range_in_axis, from: x, axis: T}

This is basically equivalent to:

x_len: {class: length, from: x}
y: {class: range_from_length, from: x_len}

Parameters:

• axis (str)

• dtype (str)

• sparse (bool)

• out_spatial_dim (Dim|None)

layer_class: Optional[str] = 'range_from_length'

recurrent = True

classmethod get_out_data_from_opts(name, sources, dtype='int32', sparse=False, out_spatial_dim=None, **kwargs)

Parameters:

• name (str)

• sources (list[LayerBase])

• dtype (str)

• sparse (bool)

• out_spatial_dim (Dim|None)

kwargs: Optional[Dict[str]]

output_before_activation: Optional[OutputWithActivation]

output_loss: Optional[tf.Tensor]

rec_vars_outputs: Dict[str, tf.Tensor]

search_choices: Optional[SearchChoices]

params: Dict[str, tf.Variable]

saveable_param_replace: Dict[tf.Variable, Union['tensorflow.python.training.saver.BaseSaverBuilder.SaveableObject', None]]

stats: Dict[str, tf.Tensor]