TFEngine

TensorFlow engine

The basic engine for the TensorFlow backend is implemented here, i.e. the high-level logic to train, i.e. looping over epochs, holding the network instance, creating the TensorFlow session, managing the data pipeline, etc.

See Technological overview for an overview how it fits all together.

exception TFEngine.CancelTrainingException[source]
class TFEngine.Runner(engine, dataset, batches, train, eval=True, extra_fetches=None, extra_fetches_callback=None)[source]
Parameters:
  • engine (Engine) –
  • dataset (Dataset.Dataset) –
  • batches (BatchSetGenerator) –
  • train (bool) – whether to do updates on the model
  • eval (bool) – whether to evaluate (i.e. calculate loss/error)
  • extra_fetches (dict[str,tf.Tensor|TFUtil.Data|TFNetworkLayer.LayerBase]|None) – additional fetches per step. extra_fetches_callback will be called with these. In case of Data/LayerBase, it will return a list, where each item corresponds to the batch-seq. It might also be useful to add network.get_extern_data(“seq_idx”) and network.get_extern_data(“seq_tag”).
  • extra_fetches_callback ((**dict[str,numpy.ndarray|str|list[numpy.ndarray|str])->None) –

    called if extra_fetches

run(report_prefix)[source]
Parameters:report_prefix (str) – prefix for logging, e.g. “train”
class TFEngine.Engine(config=None)[source]
Parameters:config (Config.Config|None) –
finalize()[source]
get_const_tensor(key, value)[source]
is_requesting_for_gpu()[source]
classmethod get_train_start_epoch_batch(config)[source]

We will always automatically determine the best start (epoch,batch) tuple based on existing model files. This ensures that the files are present and enforces that there are no old outdated files which should be ignored. Note that epochs start at idx 1 and batches at idx 0. :type config: Config.Config :returns (epoch,batch) :rtype (int,int)

classmethod config_get_final_epoch(config)[source]
Parameters:config (Config.Config) –
Return type:int
classmethod get_epoch_model(config)[source]

:returns (epoch, modelFilename) :rtype: (int|None, str|None)

classmethod epoch_model_filename(model_filename, epoch, is_pretrain)[source]
Return type:str
get_epoch_model_filename(epoch=None)[source]
get_epoch_str()[source]
is_pretrain_epoch(epoch=None)[source]
is_first_epoch_after_pretrain()[source]
get_eval_datasets()[source]
load_model(epoch=None, filename=None)[source]
Parameters:
  • epoch (int) –
  • filename (str) –
save_model(filename=None)[source]
Parameters:filename (str) – full filename for model
static delete_model(filename)[source]
Parameters:filename (str) –
Returns:accumulated file-size in bytes of deleted files
Return type:int
init_train_from_config(config=None, train_data=None, dev_data=None, eval_data=None)[source]
Parameters:
  • config (Config.Config|None) –
  • train_data (Dataset.Dataset|None) –
  • dev_data (Dataset.Dataset|None) –
  • eval_data (Dataset.Dataset|None) –
init_network_from_config(config=None)[source]
Parameters:config (Config.Config|None) –
classmethod create_network(config, rnd_seed, train_flag, eval_flag, search_flag, net_dict, initial_learning_rate=1.0)[source]
Parameters:
  • config (Config.Config) –
  • rnd_seed (int) –
  • train_flag (bool|tf.Tensor) –
  • initial_learning_rate (float) –
  • eval_flag (bool) –
  • search_flag (bool) –
  • net_dict (dict[str,dict[str]]) –
Returns:

network, updater

Return type:

(TFNetwork, Updater|None)

maybe_init_new_network(net_desc)[source]
Parameters:net_desc (dict[str,dict[str]]) – layer name -> layer description dict
train()[source]
init_train_epoch()[source]
train_epoch()[source]
format_score(score)[source]
eval_model(output_file=None)[source]

Eval the current model on the eval datasets (dev + eval, whatever is set). See also self.search() for performing beam search.

Parameters:output_file (str|None) – if given, will save the results to this file
Returns:nothing
check_last_epoch()[source]
cleanup_old_models(ask_for_confirmation=False)[source]
Parameters:ask_for_confirmation (bool) – if True, will ask the user interactively to confirm
get_all_merged_summaries()[source]
Returns:merged summaries, serialized string
Return type:tf.Tensor
check_uninitialized_vars()[source]

All vars in TF which are controlled by us should also have been initialized by us. We also take care about the optimizer slot variables. However, TF can still create other vars which we do not know about. E.g. the Adam optimizer creates the beta1_power/beta2_power vars (which are no slot vars). Here, we find all remaining uninitialized vars, report about them and initialize them.

get_specific_feed_dict(dataset, seq_idx)[source]
Parameters:
  • dataset (Dataset.Dataset) –
  • seq_idx (int) – index of sequence, -1 for all sequences in dataset
Returns:

feed_dict for self.tf_session.run()

Return type:

dict[tf.Tensor,numpy.ndarray]

run_single(dataset, seq_idx, output_dict, ext_feed_dict=None)[source]
Parameters:
  • dataset (Dataset) –
  • seq_idx (int) – index of sequence, -1 for all sequences in dataset
  • output_dict (dict[str,tf.Tensor]) – key -> tf.Tensor
  • ext_feed_dict (dict[tf.Tensor,numpy.ndarray]) –
Returns:

output_dict but values evaluated

Return type:

dict[str,numpy.ndarray]

forward_single(dataset, seq_idx, output_layer_name=None)[source]

Forwards a single sequence. If you want to perform search, and get a number of hyps out, use search_single().

Parameters:
  • dataset (Dataset.Dataset) –
  • seq_idx (int) –
  • output_layer_name (str|None) – e.g. “output”. if not set, will read from config “forward_output_layer”
Returns:

numpy array, output in time major format (time,dim)

Return type:

numpy.ndarray

forward_to_hdf(data, output_file, combine_labels='', batch_size=0)[source]

Is aiming at recreating the same interface and output as Engine.forward_to_hdf(). See also EngineTask.HDFForwardTaskThread() and hdf_dump_from_dataset() in the hdf_dump.py tool.

Parameters:
  • data (Dataset) –
  • output_file (str) –
  • combine_labels (str) – ignored at the moment
  • batch_size (int) –
analyze(data, statistics)[source]
Parameters:
  • data (Dataset.Dataset) –
  • statistics (list[str]|None) – ignored at the moment
Returns:

print everything to log.v1, and return the Runner instance to get access to all the stats

Return type:

Runner

search(dataset, do_eval=True, output_layer_name='output', output_file=None, output_file_format='txt')[source]
Parameters:
  • dataset (Dataset.Dataset) –
  • do_eval (bool) – calculate errors. can only be done if we have the reference target
  • output_layer_name (str) –
  • output_file (str) –
  • output_file_format (str) – “txt” or “py”
search_single(dataset, seq_idx, output_layer_name=None)[source]

Performs search. See also forward_single().

Parameters:
  • dataset (Dataset.Dataset) –
  • seq_idx (int) – index of sequence, -1 for all sequences in dataset
  • output_layer_name (str|None) – e.g. “output”. if not set, will read from config “search_output_layer”
Returns:

list of score and numpy array, each numpy arry in format (time,dim)

Return type:

list[(float,numpy.ndarray)]

search_single_seq(sources, output_layer_name=None)[source]
Parameters:
  • sources (list[numpy.ndarray]) – source sequences as a list of indices
  • output_layer_name (str|None) – e.g. “output”. if not set, will read from config “search_output_layer”
Returns:

list of all hyps, which is a tuple of score and string

Return type:

list[(float,str)]

search_single_string_to_string_seq(sources, output_layer_name=None)[source]
Parameters:
  • sources (str|list[str]) – source text as a string (list for batch translation)
  • output_layer_name (str|None) – e.g. “output”. if not set, will read from config “search_output_layer”
Returns:

list of all hyps, which is a tuple of score and string

Return type:

list[(float,str)]

compute_priors(dataset, config=None)[source]
Parameters:
web_server(port)[source]

Starts a web-server with a simple API to forward data through the network (or search if the flag is set).

Parameters:port (int) – for the http server
Returns:
TFEngine.get_global_engine()[source]

Similar as Config.get_global_config().

Return type:Engine