TFEngine

TensorFlow engine

The basic engine for the TensorFlow backend is implemented here, i.e. the high-level logic to train, i.e. looping over epochs, holding the network instance, creating the TensorFlow session, managing the data pipeline, etc.

See Technological overview for an overview how it fits all together.

exception TFEngine.CancelTrainingException[source]

Training was cancelled.

class TFEngine.Runner(engine, dataset, batches, train, eval=True, train_flag=None, extra_fetches=None, extra_fetches_callback=None)[source]

This encapsulates the logic around TF session.run, i.e. iterating over the dataset.

Parameters:
  • engine (Engine) –
  • dataset (Dataset.Dataset) –
  • batches (BatchSetGenerator) –
  • train (bool) – whether to do updates on the model
  • train_flag (bool|None) – normally just as train. but e.g. maybe you want to have the train_flag but not train
  • eval (bool) – whether to evaluate (i.e. calculate loss/error)
  • extra_fetches (dict[str,tf.Tensor|TFUtil.Data|TFNetworkLayer.LayerBase|()->tf.Tensor)]|None) – additional fetches per step. extra_fetches_callback will be called with these. In case of Data/LayerBase, it will return a list, where each item corresponds to the batch-seq. It might also be useful to add network.get_extern_data(“seq_idx”) and network.get_extern_data(“seq_tag”).
  • extra_fetches_callback ((**dict[str,numpy.ndarray|str|list[numpy.ndarray|str])->None) –

    called if extra_fetches

run(self, report_prefix)[source]
Parameters:report_prefix (str) – prefix for logging, e.g. “train”
class TFEngine.Engine(config=None)[source]

TF backend engine.

Parameters:config (Config.Config|None) –
finalize(self)[source]

Finalizes the TF session, network, graph.

get_const_tensor(self, key, value)[source]
Parameters:
  • key
  • value
Returns:

tf.constant(value)

Return type:

tf.Tensor

is_requesting_for_gpu(self)[source]
Return type:bool
get_eval_datasets(self)[source]
Returns:dict of datasets used for eval (dev, eval)
Return type:dict[str,Dataset]
load_model(self, epoch=None, filename=None)[source]
Parameters:
  • epoch (int) –
  • filename (str) –
save_model(self, filename=None)[source]
Parameters:filename (str) – full filename for model
static delete_model(filename)[source]
Parameters:filename (str) –
Returns:accumulated file-size in bytes of deleted files
Return type:int
init_train_from_config(self, config=None, train_data=None, dev_data=None, eval_data=None)[source]
Parameters:
  • config (Config.Config|None) –
  • train_data (Dataset.Dataset|None) –
  • dev_data (Dataset.Dataset|None) –
  • eval_data (Dataset.Dataset|None) –
init_network_from_config(self, config=None, net_dict_post_proc=None)[source]
Parameters:
  • config (Config.Config|None) –
  • net_dict_post_proc (((dict)->dict)|None) –
classmethod create_network(config, rnd_seed, train_flag, eval_flag, search_flag, net_dict, initial_learning_rate=1.0)[source]
Parameters:
  • config (Config.Config) –
  • rnd_seed (int) –
  • train_flag (bool|tf.Tensor) –
  • initial_learning_rate (float) –
  • eval_flag (bool) –
  • search_flag (bool) –
  • net_dict (dict[str,dict[str]]) –
Returns:

network, updater

Return type:

(TFNetwork, Updater|None)

need_init_new_network(self, net_desc=None)[source]
Parameters:net_desc (dict[str,dict[str]]|None) – layer name -> layer description dict
Return type:bool
init_new_network(self, net_desc=None)[source]

Reinitializes the network, and copies over the parameter from the current network.

Parameters:net_desc (dict[str,dict[str]]|None) – layer name -> layer description dict. use existing by default
train(self)[source]

Does the whole training, i.e. the loop over all the epochs.

init_train_epoch(self)[source]

Init for the current train epoch.

train_epoch(self)[source]

Train a single epoch (self.epoch).

format_score(self, score)[source]
Parameters:score (dict[str,float]) –
Returns:score(s) as str
Return type:str
eval_model(self, output_file=None, output_per_seq_file=None, loss_name=None, output_per_seq_format=None, output_per_seq_file_format='txt')[source]

Eval the current model on the eval datasets (dev + eval, whatever is set). See also self.search() for performing beam search.

Parameters:
  • output_file (str|None) – if given, will save the results to this file (total err/score for each dataset)
  • output_per_seq_file (str|None) – if given, will save the err/score for each sequence
  • loss_name (str|None) – specifies the loss which will be written to output_file
  • output_per_seq_format (list[str]|tuple[str]|None) – which properties of loss_name should be written to output_per_seq_file. allowed_outputs = {“seq_tag”, “seq_len”, “score”, “error”, “pos_score”, “pos_error”}.
  • output_per_seq_file_format (str) – “txt” or “py”
Returns:

nothing

check_last_epoch(self)[source]

Checks if there are outstanding tasks (eval_model) for the last epoch, and executes them.

cleanup_old_models(self, ask_for_confirmation=False)[source]
Parameters:ask_for_confirmation (bool) – if True, will ask the user interactively to confirm
check_uninitialized_vars(self)[source]

All vars in TF which are controlled by us should also have been initialized by us. We also take care about the optimizer slot variables. However, TF can still create other vars which we do not know about. E.g. the Adam optimizer creates the beta1_power/beta2_power vars (which are no slot vars). Here, we find all remaining uninitialized vars, report about them and initialize them.

get_specific_feed_dict(self, dataset, seq_idx)[source]
Parameters:
  • dataset (Dataset.Dataset) –
  • seq_idx (int) – index of sequence, -1 for all sequences in dataset
Returns:

feed_dict for self.tf_session.run()

Return type:

dict[tf.Tensor,numpy.ndarray]

run_single(self, dataset, seq_idx, output_dict, ext_feed_dict=None)[source]
Parameters:
  • dataset (Dataset) –
  • seq_idx (int) – index of sequence, -1 for all sequences in dataset
  • output_dict (dict[str,tf.Tensor]) – key -> tf.Tensor
  • ext_feed_dict (dict[tf.Tensor,numpy.ndarray]) –
Returns:

output_dict but values evaluated

Return type:

dict[str,numpy.ndarray]

forward_single(self, dataset, seq_idx, output_layer_name=None)[source]

Forwards a single sequence. If you want to perform search, and get a number of hyps out, use search_single().

Parameters:
  • dataset (Dataset.Dataset) –
  • seq_idx (int) –
  • output_layer_name (str|None) – e.g. “output”. if not set, will read from config “forward_output_layer”
Returns:

numpy array, output in time major format (time,dim)

Return type:

numpy.ndarray

forward_to_hdf(self, data, output_file, combine_labels='', batch_size=0, output_layer=None)[source]

Is aiming at recreating the same interface and output as Engine.forward_to_hdf(). See also EngineTask.HDFForwardTaskThread() and hdf_dump_from_dataset() in the hdf_dump.py tool.

Parameters:
  • data (Dataset) –
  • output_file (str) –
  • combine_labels (str) – ignored at the moment
  • batch_size (int) –
  • output_layer (LayerBase) –
analyze(self, data, statistics)[source]
Parameters:
  • data (Dataset.Dataset) –
  • statistics (list[str]|None) – ignored at the moment
Returns:

print everything to log.v1, and return the Runner instance to get access to all the stats

Return type:

Runner

search(self, dataset, do_eval=True, output_layer_names='output', output_file=None, output_file_format='txt')[source]
Parameters:
  • dataset (Dataset.Dataset) –
  • do_eval (bool) – calculate errors. can only be done if we have the reference target
  • output_layer_names (str|list[str]) –
  • output_file (str) –
  • output_file_format (str) – “txt” or “py”
search_single(self, dataset, seq_idx, output_layer_name=None)[source]

Performs search. See also forward_single().

Parameters:
  • dataset (Dataset.Dataset) –
  • seq_idx (int) – index of sequence, -1 for all sequences in dataset
  • output_layer_name (str|None) – e.g. “output”. if not set, will read from config “search_output_layer”
Returns:

list of score and numpy array, each numpy arry in format (time,dim)

Return type:

list[(float,numpy.ndarray)]

search_single_seq(self, sources, output_layer_name=None)[source]
Parameters:
  • sources (list[numpy.ndarray]) – source sequences as a list of indices
  • output_layer_name (str|None) – e.g. “output”. if not set, will read from config “search_output_layer”
Returns:

list of all hyps, which is a tuple of score and string

Return type:

list[(float,str)]

search_single_string_to_string_seq(self, sources, output_layer_name=None)[source]
Parameters:
  • sources (str|list[str]) – source text as a string (list for batch translation)
  • output_layer_name (str|None) – e.g. “output”. if not set, will read from config “search_output_layer”
Returns:

list of all hyps, which is a tuple of score and string

Return type:

list[(float,str)]

compute_priors(self, dataset, config=None)[source]
Parameters:
web_server(self, port)[source]

Starts a web-server with a simple API to forward data through the network (or search if the flag is set).

Parameters:port (int) – for the http server
Returns:
TFEngine.get_global_engine()[source]

Similar to Config.get_global_config().

Return type:Engine