Utility Layers#
Framewise Statistics Layer#
- class returnn.tf.layers.basic.FramewiseStatisticsLayer(sil_label_idx, histogram_num_bins=20, **kwargs)[source]#
Collects various statistics (such as FER, etc) on the sources. The tensors will get stored in self.stats which will be collected by TFEngine.
Usually the arguments, when specified in the network dict, are going through
transform_config_dict()
, before they are passed to here. SeeTFNetwork.construct_from_dict()
.- Parameters:
name (str) –
network (returnn.tf.network.TFNetwork) –
output (Data) – Set a specific output instead of using
get_out_data_from_opts()
n_out (NotSpecified|None|int) – output dim
out_dim (returnn.tensor.Dim|None) – output feature dim tag
out_type (dict[str]) – kwargs for Data class. more explicit than n_out.
out_shape (set[returnn.tensor.Dim|returnn.tf.util.data._MarkedDim]|tuple|list|None) – verifies the output shape (dim tags). See
Data.verify_out_shape()
.sources (list[LayerBase]) – via self.transform_config_dict()
in_dim (returnn.tensor.Dim|None) – input feature dim tag
target (str|list[str]|None) – if some loss is set, this is the target data-key, i.e. network.extern_data.get_data(target). alternatively, this also can be a layer name.
_target_layers (dict[str,LayerBase]|None) – if target.startswith(“layer:”), then this is target -> layer
size_target (str|None) – like target but this is only used to set our output size in case of training
loss (Loss|None) – via
transform_config_dict()
. Every layer can have one loss (of typeLoss
), or none loss. In the net dict, it is specified as a string. InTFNetwork
, all losses from all layers will be collected. That is whatTFUpdater.Updater
will use for training.reuse_params (ReuseParams|None) – if given, will opt reuse the params. see
self.var_creation_scope()
. See also thename_scope
option as an alternative.name_scope (str|None) – If set, uses this custom (relative) name scope. If it starts with a “/”, it will be the absolute name scope. It should not end with a “/”. It can be empty, in which case it will not consume a new name scope. This can also be used for parameter sharing. The default is the layer name in most cases, but this logic is in
get_absolute_name_scope_prefix()
andTFNetwork.layer_creation_scope()
.param_device (str|None) – e.g. “CPU”, etc. any valid name for tf.device. see https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/util/device_name_utils.h
L2 (float|None) – for constraints
darc1 (float|None) – for constraints. see Generalization in Deep Learning, https://arxiv.org/abs/1710.05468
spatial_smoothing (float|None) – see
returnn.tf.util.basic.spatial_smoothing_energy()
param_variational_noise (float|None) – adds variational noise to the params during training
param_dropout (float|None) – dropout on params (weight dropout) during training
param_dropout_min_ndim (int|None) – if param dropout is enabled, only use if for params whose ndim >= this. E.g. it might make sense to disable it for bias params or scalars, so set param_dropout_min_ndim=2.
updater_opts (dict[str]|None) – accepts similar opts as TFUpdater, e.g. “optimizer”, “learning_rate”, …
is_output_layer (bool|None) – triggers the construction of this layer in the root net. Inside a
RecLayer
, it triggers the explicit accumulation of all frames. Also see theneed_last
option.only_on_eval (bool) – if True, this layer will only be calculated in eval
only_on_search (bool) – if True, this layer will only be calculated when search is done
copy_output_loss_from_source_idx (int|None) – if set, will copy output_loss from this source
batch_norm (bool|dict) – see self.batch_norm()
initial_output (str|float) – used for recurrent layer, see self.get_rec_initial_output()
state – explicitly defines the rec state. initial_state would define the initial state (in the first frame)
need_last (bool) – Inside
RecLayer
, make sure that we can access the last frame. Similar to ``is_output_layer, but this is specifically about the last frame, i.e. it does not trigger accumulation.rec_previous_layer (LayerBase|None) – via the recurrent layer, layer (template) which represents the past of us. You would not explicitly set this in a config. This is automatically, internally, via
RecLayer
.encapsulate (bool) –
mostly relevant for SubnetworkLayer and similar: If True, all sub layers will be created,
and covered in functions like
get_rec_initial_extra_outputs()
, and the logic incls_get_sub_network()
will not be used.If False, the logic in
cls_get_sub_network()
will be used.collocate_with (list[str]|None) – in the rec layer, collocate with the specified other layers
trainable (bool) – whether the parameters of this layer will be trained. Default is True. However, if this is inside a subnetwork, all the parent layers must be set to trainable, otherwise the parameters will not be trainable.
custom_param_importer (str|callable|None) – used by
set_param_values_by_dict()
register_as_extern_data (str|None) – registers output in network.extern_data
control_dependencies_on_output (None|((LayerBase)->list[tf.Operation])) – This is mostly to perform some checks after the layer output has been computed, before the layer output is used anywhere else. There is also the
IdentityLayer
with the optioncontrol_dependencies
.debug_print_layer_output (None|bool|dict[str]) – same as global config option but per layer
_name (str) – just for internal construction, should be the same as
name
_network (returnn.tf.network.TFNetwork) – just for internal construction, should be the same as
network
_src_common_search_choices (None|SearchChoices) – set via
SearchChoices.translate_to_common_search_beam()
- output_before_activation: Optional[OutputWithActivation][source]#
- search_choices: Optional[SearchChoices][source]#
HDFDumpLayer#
- class returnn.tf.layers.basic.HDFDumpLayer(filename, extra=None, dump_whole_batches=False, labels=None, extend_existing_file=False, dump_per_run=False, **kwargs)[source]#
Dumps into HDF file, compatible to
HDFDataset
.The HDF will be written to disk under the specified filename, if there was no error, by default at graph reset, via
TFNetwork.register_graph_reset_callback()
. Or after the dataset iteration run loop, with dump_per_run, viaTFNetwork.register_run_finished_callback()
.Common usage would be to add this to your network with “is_output_layer”: True, such that you don’t need to make other layers depend on it.
It currently uses
SimpleHDFWriter
internally.- Parameters:
filename (str|(()->str)) –
extra (None|dict[str,LayerBase]) –
dump_whole_batches (bool) – dumps the whole batch as a single sequence into the HDF
labels (list[str]|None) –
extend_existing_file (bool) – True also means we expect that it exists
dump_per_run (bool) – write via
TFNetwork.register_run_finished_callback()
- classmethod get_out_data_from_opts(name, sources, **kwargs)[source]#
- Parameters:
name (str) –
sources (list[LayerBase]) –
- Return type:
Data
- classmethod transform_config_dict(d, network, get_layer)[source]#
- Parameters:
d (dict[str]) – will modify inplace
network (returnn.tf.network.TFNetwork) –
get_layer (((str) -> LayerBase)) – function to get or construct another layer
- output_before_activation: Optional[OutputWithActivation][source]#
- search_choices: Optional[SearchChoices][source]#
Image Summary Layer#
- class returnn.tf.layers.basic.ImageSummaryLayer(max_outputs=3, **kwargs)[source]#
Creates image summaries which can be viewed in TensorBoard. This layer expects the source to be in (T-decoder, T-encoder, B, 1).
- Parameters:
max_outputs – number of images to generate per step
- classmethod transform_config_dict(d, network, get_layer)[source]#
- Parameters:
d (dict[str]) – will modify inplace, the loss_opts
network (returnn.tf.network.TFNetwork) –
get_layer (((str) -> LayerBase)) – function to get or construct another layer
- output_before_activation: Optional[OutputWithActivation][source]#
- search_choices: Optional[SearchChoices][source]#
Print Layer#
- class returnn.tf.layers.basic.PrintLayer(summarize=99, extra_print_args=(), **kwargs)[source]#
Prints the sources to console/log, via
returnn.tf.util.basic.py_print()
.- Parameters:
summarize (int|None) – passed to
py_print()
extra_print_args (list|tuple) –
- classmethod get_out_data_from_opts(name, sources, **kwargs)[source]#
- Parameters:
name (str) –
sources (list[LayerBase]) –
- Return type:
Data
- output_before_activation: Optional[OutputWithActivation][source]#
- search_choices: Optional[SearchChoices][source]#
Scaled Gradient Layer#
- class returnn.tf.layers.basic.ScaledGradientLayer(scale, shift=None, scale_shift_by_sum_over_axis=None, clip_max_axis=None, **kwargs)[source]#
Just
tf.identity()
in the forward pass. Scales the gradient by some factor in backprop. Can be used as gradient reversal layer (with negative factor). Usesreturnn.tf.util.basic.scaled_gradient()
, ortf.stop_gradient()
- Parameters:
scale (float|LayerBase) – if 0. and no shift, will use tf.stop_gradient
shift (float|LayerBase|None) –
scale_shift_by_sum_over_axis (Dim|str|None) – if given, calculates the sum over this axis (absolute values) and multiplies the shift value by this sum.
clip_max_axis (Dim|str|None) – if given, clips the gradient to the max value in this axis before the transformation, for all values in the axis
- classmethod transform_config_dict(d, network, get_layer)[source]#
- Parameters:
d (dict[str]) – will modify inplace
network (returnn.tf.network.TFNetwork) –
get_layer (((str) -> LayerBase)) – function to get or construct another layer
- output_before_activation: Optional[OutputWithActivation][source]#
- search_choices: Optional[SearchChoices][source]#
Synthetic Gradient Layer#
- class returnn.tf.layers.basic.SyntheticGradientLayer(gradient, meta_loss_scale=1.0, **kwargs)[source]#
This is a generalized way to be able to replace the true gradient with any kind of predicted gradient. This enabled to implement the idea from here:
Decoupled Neural Interfaces using Synthetic Gradients, https://arxiv.org/abs/1608.05343
- Parameters:
gradient (LayerBase) –
meta_loss_scale (float) –
- classmethod transform_config_dict(d, network, get_layer)[source]#
- Parameters:
d (dict[str]) –
network (returnn.tf.network.TFNetwork) –
get_layer –
- classmethod get_out_data_from_opts(sources, name, **kwargs)[source]#
- Parameters:
sources (list[LayerBase]) –
name (str) –
- Return type:
Data
- output_before_activation: Optional[OutputWithActivation][source]#
- search_choices: Optional[SearchChoices][source]#