Generative Layers#

Positional Encoding Layer#

class returnn.tf.layers.rec.PositionalEncodingLayer(axis=<class 'returnn.util.basic.NotSpecified'>, add_to_input=False, constant=-1, offset=None, **kwargs)[source]#

Provides positional encoding in the form of (batch, time, n_out) or (time, batch, n_out) where n_out is the number of channels, if it is run outside a RecLayer, and (batch, n_out) or (n_out, batch) if run inside a RecLayer, where it will depend on the current time frame.

Assumes one source input with a time dimension if outside a RecLayer. With add_to_input, it will calculate x + input, and the output shape is the same as the input

The positional encoding is the same as in Tensor2Tensor. See returnn.tf.util.basic.get_positional_encoding().

Parameters:
  • axis (Dim|str|NotSpecified) – if not specified, check for time_dim_axis, otherwise assume rec step

  • add_to_input (bool) – will add the signal to the input

  • constant (int) – if positive, always output the corresponding positional encoding.

  • offset (None|LayerBase) – Specify the offset to be added to positions. Expect shape (batch, time) or (batch,).

layer_class: Optional[str] = 'positional_encoding'[source]#
recurrent = True[source]#
classmethod transform_config_dict(d, network, get_layer)[source]#
Parameters:
classmethod get_out_data_from_opts(name, network, add_to_input=False, sources=(), **kwargs)[source]#
Parameters:
Return type:

Data

classmethod get_rec_initial_extra_outputs(batch_dim, rec_layer, network, **kwargs)[source]#
Parameters:
Return type:

dict[str,tf.Tensor]

classmethod get_rec_initial_extra_outputs_shape_invariants(rec_layer, network, **kwargs)[source]#
Parameters:
Returns:

optional shapes for the tensors by get_rec_initial_extra_outputs

Return type:

dict[str,tf.TensorShape]

kwargs: Optional[Dict[str]][source]#
output_before_activation: Optional[OutputWithActivation][source]#
output_loss: Optional[tf.Tensor][source]#
rec_vars_outputs: Dict[str, tf.Tensor][source]#
search_choices: Optional[SearchChoices][source]#
params: Dict[str, tf.Variable][source]#
saveable_param_replace: Dict[tf.Variable, Union['tensorflow.python.training.saver.BaseSaverBuilder.SaveableObject', None]][source]#
stats: Dict[str, tf.Tensor][source]#
input_data: Optional[Data][source]#

Range Layer#

class returnn.tf.layers.basic.RangeLayer(limit, start=0, delta=1, dtype=None, sparse=False, out_spatial_dim=None, **kwargs)[source]#

Generic wrapper around tf.range. See also RangeInAxisLayer.

Parameters:
  • limit (int|float) –

  • start (int|float) –

  • delta (int|float) –

  • dtype (str|None) –

  • sparse (bool) –

  • out_spatial_dim (Dim|None) –

layer_class: Optional[str] = 'range'[source]#
classmethod transform_config_dict(d, network, get_layer)[source]#
Parameters:
classmethod get_out_data_from_opts(name, limit, start=0, delta=1, dtype=None, sparse=False, out_spatial_dim=None, **kwargs)[source]#
Parameters:
  • name (str) –

  • limit (int|float) –

  • start (int|float) –

  • delta (int|float) –

  • dtype (str|None) –

  • sparse (bool) –

  • out_spatial_dim (Dim|None) –

Return type:

Data

kwargs: Optional[Dict[str]][source]#
output_before_activation: Optional[OutputWithActivation][source]#
output_loss: Optional[tf.Tensor][source]#
rec_vars_outputs: Dict[str, tf.Tensor][source]#
search_choices: Optional[SearchChoices][source]#
params: Dict[str, tf.Variable][source]#
saveable_param_replace: Dict[tf.Variable, Union['tensorflow.python.training.saver.BaseSaverBuilder.SaveableObject', None]][source]#
stats: Dict[str, tf.Tensor][source]#

Range in Axis Layer#

class returnn.tf.layers.basic.RangeInAxisLayer(axis, dtype='int32', unbroadcast=False, keepdims=False, sparse=False, **kwargs)[source]#

Assume that the input is e.g. (B,T,D), and you specify axis=”T”, you will get (T,), where the specified axis is filled with tf.range. See also RangeLayer.

Parameters:
  • axis (str|Dim) –

  • dtype (str) –

  • unbroadcast (bool) – DEPRECATED, unsupported, and not needed

  • keepdims (bool) – DEPRECATED, unsupported, and not needed

  • sparse (bool) –

layer_class: Optional[str] = 'range_in_axis'[source]#
recurrent = True[source]#
classmethod get_out_data_from_opts(name, sources, axis, dtype='int32', sparse=False, **kwargs)[source]#
Parameters:
  • name (str) –

  • sources (list[LayerBase]) –

  • axis (str|Dim) –

  • dtype (str) –

  • sparse (bool) –

kwargs: Optional[Dict[str]][source]#
output_before_activation: Optional[OutputWithActivation][source]#
output_loss: Optional[tf.Tensor][source]#
rec_vars_outputs: Dict[str, tf.Tensor][source]#
search_choices: Optional[SearchChoices][source]#
params: Dict[str, tf.Variable][source]#
saveable_param_replace: Dict[tf.Variable, Union['tensorflow.python.training.saver.BaseSaverBuilder.SaveableObject', None]][source]#
stats: Dict[str, tf.Tensor][source]#

Range from Length Layer#

class returnn.tf.layers.basic.RangeFromLengthLayer(dtype='int32', sparse=False, out_spatial_dim=None, **kwargs)[source]#

Given some dynamic sequence lengths as input, this creates a tf.range over the implied dimension. As a side effect, this can create a new dyn dim tag for the given sequence lengths. This side effect can be the main functionality in certain use cases. See also RangeInAxisLayer.

Consider the example:

y: {class: range_in_axis, from: x, axis: T}

This is basically equivalent to:

x_len: {class: length, from: x}
y: {class: range_from_length, from: x_len}
Parameters:
  • axis (str) –

  • dtype (str) –

  • sparse (bool) –

  • out_spatial_dim (Dim|None) –

layer_class: Optional[str] = 'range_from_length'[source]#
recurrent = True[source]#
classmethod get_out_data_from_opts(name, sources, dtype='int32', sparse=False, out_spatial_dim=None, **kwargs)[source]#
Parameters:
  • name (str) –

  • sources (list[LayerBase]) –

  • dtype (str) –

  • sparse (bool) –

  • out_spatial_dim (Dim|None) –

kwargs: Optional[Dict[str]][source]#
output_before_activation: Optional[OutputWithActivation][source]#
output_loss: Optional[tf.Tensor][source]#
rec_vars_outputs: Dict[str, tf.Tensor][source]#
search_choices: Optional[SearchChoices][source]#
params: Dict[str, tf.Variable][source]#
saveable_param_replace: Dict[tf.Variable, Union['tensorflow.python.training.saver.BaseSaverBuilder.SaveableObject', None]][source]#
stats: Dict[str, tf.Tensor][source]#