Custom Layers

Eval Layer

class, **kwargs)[source]

Evaluates some string. The CombineLayer provides this functionality, thus this is just a special case of it. Also see ActivationLayer, or CompareLayer.

The output type is defined as a broadcasted extension of all sources. You can overwrite it by (partially) specifying out_type. out_type can also be a generic Python function, returning a Data instance.

Parameters:eval (str) – will eval this string. see _op_kind_eval()
layer_class = 'eval'[source]

Subnetwork Layer

class, _subnet, _output, concat_sources=True, load_on_init=None, dropout=0, dropout_noise_shape=None, _parent_layer_cache=None, _from=None, **kwargs)[source]

You can define a whole subnetwork as a single layer by this class.

The subnetwork will be specified by a dict[str,dict[str]], just like a normal network is specified in the config.

The "output" layer of the subnetwork will be the output of this subnetwork-layer.

With concat_sources=True (default),
the input to this layer will be represented as the "data:data" or simply "data" in the subnetwork,
otherwise with concat_sources=False,
the input to this layer will be represented as "data:input_layer_name" and also "data:0" to "data:<n-1>" for n inputs, for each input, in the subnetwork. The first input will also be simply available as "data:data"/``”data”`.
  • subnetwork (dict[str,dict]) – subnetwork as dict (JSON content). must have an “output” layer-
  • concat_sources (bool) – if we concatenate all sources into one, like it is standard for most other layers
  • load_on_init (str|dict[str]|None) – if provided, for parameter initialization, we will load the given model file. see CustomCheckpointLoader.
  • dropout (float) – will be applied if train_flag is set
  • dropout_noise_shape (tuple|list|dict|None) –
  • _parent_layer_cache (dict[str,LayerBase]|None) –
  • _subnet ( –
  • _output (LayerBase) –
layer_class = 'subnetwork'[source]
recurrent = True[source]

Update self.params.


Update self.rec_vars_outputs.


Handle load_on_init.

classmethod get_out_data_from_opts(n_out=<class 'returnn.util.basic.NotSpecified'>, out_type=None, **kwargs)[source]
  • n_out (int|None|NotSpecified) –
  • out_type (dict[str]|None) –
Return type:


classmethod transform_config_dict(d, network, get_layer)[source]
classmethod cls_get_sub_network(name, network, layer_desc)[source]
Return type:|None

Parameters:layer_name (str) – name of the sub_layer (right part of ‘/’ separated path)
Returns:the sub_layer addressed in layer_name or None if no sub_layer exists
Return type:LayerBase|None
Return type:list[]
Return type:list[LayerBase]
Returns:list of layers this layer depends on. normally this is just self.sources but e.g. the attention layer in addition has a base, etc.
Return type:list[LayerBase]
Parameters:key (int|str|None) – also the special key “*”
Return type:tf.Tensor|None
classmethod get_rec_initial_extra_outputs(batch_dim, rec_layer, **kwargs)[source]
Return type:


classmethod get_rec_initial_extra_outputs_shape_invariants(**kwargs)[source]
Returns:optional shapes for the tensors by get_rec_initial_extra_outputs
Return type:dict[str,tf.TensorShape]