Updater

class Updater.Updater(momentum=0.0, nesterov_momentum=0.0, momentum2=0.0, gradient_clip=-1.0, update_clip=-1.0, weight_clip=0.0, adagrad=False, adadelta=False, adadelta_decay=0.9, adadelta_offset=1e-06, max_norm=0.0, adasecant=False, adam=False, adamdelta=False, adam_fit_learning_rate=True, adamax=False, nadam=False, nadam_decay=0.004, eve=False, gradient_l2_norm=False, mean_normalized_sgd=False, mean_normalized_sgd_average_interpolation=0.5, rmsprop=0.0, smorms3=False, update_multiple_models=0, update_multiple_models_average_step=0, update_multiple_models_average_step_i=0, update_multiple_models_averaging=True, update_multiple_models_param_is_cur_model=False, multi_batch_update=0, variance_reduction=False, enforce_triangular_matrix_zero=False, gradient_noise=0.0, gradient_noise_decay=0.55, grad_noise_rel_grad_norm=0.0, reset_update_params=False)[source]

This defines how to update the model parameters per mini-batch. All kind of gradient-based optimization methods are implemented here, such as Adam etc.

Initializes the Updater class. All the params determine the specific optimization variants and their hyper params. Normally this is constructed by Updater.initFromConfig().

Parameters:
  • momentum
  • nesterov_momentum
  • momentum2
  • gradient_clip
  • update_clip
  • adagrad
  • adadelta
  • adadelta_decay
  • adadelta_offset
  • max_norm
  • adasecant
  • adam
  • adamdelta
  • adam_fit_learning_rate
  • adamax
  • eve – Eve optimizer - Adam with a feedback from loss
  • adamvr – Adam with adasecant variance reduction
  • nadam – Adam with nag part momentum
  • nadam_decay
  • mean_normalized_sgd
  • mean_normalized_sgd_average_interpolation
  • rmsprop
  • smorms3
  • update_multiple_models
  • update_multiple_models_average_step
  • update_multiple_models_average_step_i
  • update_multiple_models_averaging
  • update_multiple_models_param_is_cur_model
  • multi_batch_update
  • enforce_triangular_matrix_zero
  • gradient_noise
  • gradient_noise_decay
  • grad_noise_rel_grad_norm
  • reset_update_params
getUpdateList()[source]
classmethod initFromConfig(config)[source]

Will construct a Updater instance where all params are automatically determined by the given config.

Parameters:config (Config.Config) –
Return type:Updater
classmethod initRule(rule, **kwargs)[source]
initVars(network, net_param_deltas)[source]

Initializes the Theano shared variables. This should be called in the process where you want to do the updating. All further calls must be from the same process. The network.gparams must be created in the same process.

isInitialized[source]
norm_constraint(tensor_var, max_norm, norm_axes=None, epsilon=1e-12)[source]
reset()[source]
setLearningRate(learning_rate)[source]
setNetParamDeltas(net_param_deltas)[source]
update()[source]
var(value, name='', broadcastable=None, dtype='float32', zero=False)[source]