returnn.pretrain

This module contains pretrain related code. This is independent from the backend (TF or Theano, etc).

class returnn.pretrain.WrapEpochValue(func)

Use this wrapper if you want to define some value in your network which depends on the pretrain epoch. This is going to be part in your network description dict.

Parameters:

((epoch – int) -> object) func: function which should accept one kwd-arg ‘epoch’

get_value(epoch)

Parameters:

epoch (int)

Returns:

anything, whatever self.func returns

Return type:

object

returnn.pretrain.find_pretrain_wrap_values(net_json)

See also Pretrain._resolve_wrapped_values(). Recursively goes through dicts, tuples and lists. This is a simple check to see if this is needed, i.e. if there are any WrapEpochValue used.

Parameters:

net_json (dict[str]) – network dict

Returns:

whether there is some WrapEpochValue in it

Return type:

bool

class returnn.pretrain.Pretrain(original_network_json, network_init_args=None, copy_param_mode=None, copy_output_layer=None, greedy=None, repetitions=None, construction_algo='from_output', output_layers=('output',), input_layers=('data',))

Start with 1 hidden layers up to N hidden layers -> N pretrain steps -> N epochs (with repetitions == 1). The first hidden layer is the input layer. This works for generic network constructions. See _construct_epoch().

Parameters:

• network_init_args (dict[str]|None) – additional args we use for LayerNetwork.from_json(). must have n_in, n_out. (for Theano only, thus optional now)

• copy_param_mode (str)

• copy_output_layer (bool|str) – whether to copy the output layer params from last epoch or reinit

• greedy (bool) – if True, only train output+last layer, otherwise train all

• repetitions (None | int | list[int] | dict) – how often to repeat certain pretrain steps. default is one epoch. It can also be a dict, with keys like ‘default’ and ‘final’. See code below.

• construction_algo (str|callable) – e.g. “from_output”

• output_layers (list[str]|tuple[str]) – used for construction

• input_layers (list[str]|tuple[str]) – used for construction

get_train_num_epochs()

Return type:

int

get_final_network_json()

Return type:

dict[str,dict[str]]

get_network_json_for_epoch(epoch)

Parameters:

epoch (int) – starting at 1

Return type:

dict[str]

get_train_param_args_for_epoch(epoch)

:returns the kwargs for LayerNetwork.set_train_params, i.e. which params to train. :rtype: dict[str]

returnn.pretrain.pretrain_from_config(config)

Return type:

Pretrain | None

returnn.pretrain.demo()

Will print out the different network topologies of the specified pretraining scheme.