TFUtil

class TFUtil.CollectionKeys[source]

Extension of tf.GraphKeys

RETURNN_LAYERS = '_RETURNN_layers'[source]
RETURNN_NET_STACK = '_RETURNN_network_stack'[source]
STATE_VARS = '_RETURNN_state_vars'[source]
TFUtil.tf_version_tuple()[source]
Returns:version tuple, e.g. (1, 1, 0), parsed from tf.__version__
Return type:tuple[int]
TFUtil.assert_min_tf_version(version, reason)[source]
Parameters:
  • version (tuple[int]) – e.g. (1,2,0) or (1,2)
  • reason (str) –
TFUtil.have_min_tf_version(version)[source]
Parameters:version (tuple[int]) – e.g. (1,2,0) or (1,2)
Returns:True if we have at least that version, or newer
Return type:bool
class TFUtil.Data(name, shape=None, dtype=None, placeholder=None, sparse=None, dim=None, size_placeholder=None, batch_dim_axis=0, time_dim_axis=<class 'Util.NotSpecified'>, available_for_inference=True, auto_create_placeholders=False, vocab=None, beam_size=None)[source]

This class is to describe a tensor, i.e. it’s shape and properties like whether we should consider it as sparse data (i.e. it represents indices). This is used in TFNetwork to describe the dataset external data as well as for every layer output.

Parameters:
  • name (str) –
  • shape (tuple[int|None]|list[int|None]) – including time-dim (can be None). excluding batch-dim. e.g. (time,feat)=(None,128)
  • dtype (str) – e.g. “float32” or “int64”
  • placeholder (tf.Tensor|None) – with added batch-dim
  • sparse (bool) – whether to treat the value as an index. do not confuse with tf.SparseTensor
  • dim (None|int) – feature dimension, shape[-1] if not sparse, otherwise like num_classes
  • batch_dim_axis (int|None) – where we add the batch-dim. e.g. shape=(time,…), 0 -> (batch,time,…), 1 -> (time,batch,…). This is normally always set, and a lot of code expects this. However, you can set it to None if this Data does not have a batch-dim.
  • time_dim_axis (int|None) – where we have the time dim axis, after we added the batch-dim. this is often 1. however, can be None if there is no time-dim.
  • tf.Tensor size_placeholder (dict[int,tf.Tensor]) – for every None in shape, this will describe the size. The size is always a tensor of shape (batch,), i.e. the size can be different for each sequence in a batch.
  • available_for_inference (bool) – e.g. the extern data “classes” is usually not available for inference
  • vocab (str|dict[str]|GeneratingDataset.Vocabulary|None) –
  • beam_size (int|None) – the batch-dim could be extended by a beam-size, such that it represents the merged dims [batch, beam_size].
size_dtype = 'int32'[source]
get_placeholder_kwargs(with_batch=True)[source]
get_axes_with_size()[source]
Returns:list of axes which can vary in size for each entry of the batch-dim, e.g. the time-dim-axis. The axis index is counted without the batch-dim.
Return type:list[int]
get_size_placeholder_kwargs(axis, with_batch=True)[source]
get_kwargs()[source]
get_description(with_name=True, with_placeholder=False)[source]
copy(name=None)[source]
Parameters:name (str) – if given, will overwrite this name
Returns:copy of myself, using self.get_kwargs(), and with placeholder and size_placeholder
Return type:Data
copy_as_batch_major()[source]
Returns:copy of myself with batch_dim_axis == 0
Return type:Data
copy_as_time_major()[source]
Returns:copy of myself with time_dim_axis == 0
Return type:Data
copy_with_batch_dim_axis(batch_dim_axis)[source]
Parameters:batch_dim_axis (int) –
Returns:copy of myself with specific batch_dim_axis
Return type:Data
copy_with_time_dim_axis(time_dim_axis)[source]
Parameters:time_dim_axis (int) –
Returns:copy of myself with specific time_dim_axis
Return type:Data
copy_as_bt_or_tb_major()[source]
Return type:Data
Returns:copy of myself in batch-time-major or time-batch-major
copy_add_batch_dim(batch_dim_axis)[source]
Parameters:batch_dim_axis (int) –
Returns:copy of myself with added batch-dim
Return type:Data
copy_add_spatial_dim(spatial_dim_axis)[source]
Parameters:spatial_dim_axis (int) – counted with batch-dim. if there is no time-dim, this will be it
Returns:copy of myself with added spatial-dim
Return type:Data
copy_compatible_to(data, unbroadcast=False, data_dyn_shape=None)[source]
Parameters:
  • data (Data) – other data which the returned tensor should be compatible to It would add any missing axes with a dim 1 axis for automatic broadcasting. It currently does not check whether existing dims match.
  • unbroadcast (bool) – if True, all broadcast axes (axes with dim 1) will be tiled such that they match
  • data_dyn_shape (tf.Tensor|list[tf.Tensor|int]|tuple[tf.Tensor|int]|None) – For unbroadcast, if we do not want to rely on tf.shape(data.placeholder).
Returns:

Data, might add broadcast dimensions

Return type:

Data

copy_time_flattened()[source]
Returns:copy of myself where the time-axis is flattened away into the batch-dim-axis. See get_placeholder_time_flattened() and :func:`flatten_with_seq_len_mask for more details.
Return type:Data
copy_extend_with_beam(beam_size, dyn_beam_size=None)[source]
Parameters:
  • beam_size (int) –
  • dyn_beam_size (tf.Tensor|None) – if the beam size is dynamic
Returns:

copy of myself where the batch-dim is extended/multiplied by beam_size, using tile_transposed

Return type:

Data

copy_template(name=None)[source]
Returns:copy of myself, using self.get_kwargs(), without placeholder
Return type:Data
copy_template_excluding_time_dim(name=None)[source]
Parameters:name (str|None) – if set, this will be the new name
Returns:copy of myself excluding the time-dimension without placeholder
Return type:Data
copy_template_adding_time_dim(name=None, time_dim_axis=0)[source]

Adds a time-dim-axis. If a time-dim-axis already exists, it will anyway create this new one.

Parameters:
  • name (str|None) – if set, this will be the new name
  • time_dim_axis (int) – the new time-dim-axis index
Returns:

copy of myself adding the time-dimension without placeholder

Return type:

Data

matches_var_dim_pattern(other)[source]
Parameters:other (Data) –
Returns:whether the variable-dims pattern matches, i.e. same variable dims (get_variable_dim_pattern), same time dim, excluding batch-dim. i.e. the size_placeholder should be compatible.
Return type:bool
batch_shape[source]
Returns:shape with added batch-dim. e.g. (batch,time,feat) = (None,None,128)
Return type:tuple[int|None]
get_batch_shape(batch_dim)[source]
Parameters:batch_dim (int|tf.Tensor|None) –
Returns:shape with added batch-dim. e.g. (batch,time,feat) = (None,None,128)
Return type:tuple[int|None]
shape_dense[source]
batch_shape_dense[source]
ndim[source]
Return type:int
Returns:ndim counted without batch-dim
ndim_dense[source]
Return type:int
Returns:ndim counted without batch-dim, added by 1 if we are sparse
batch_ndim[source]
Return type:int
Returns:ndim counted with batch-dim
batch_ndim_dense[source]
Return type:int
Returns:ndim counted with batch-dim, added by 1 if we are sparse
is_time_major[source]
Returns:whether this is in time-major format, i.e. (time,batch,…)
Return type:bool
is_batch_major[source]
Returns:whether this is in batch-major format, i.e. (batch,…)
Return type:bool
time_dim_axis_excluding_batch[source]
time_dimension()[source]
Returns:shape(placeholder)[time_dim_axis], int scalar
Return type:tf.Tensor
get_placeholder_as_time_major()[source]
get_placeholder_as_batch_major()[source]
get_placeholder_with_specific_batch_dim_axis(batch_dim_axis)[source]
get_placeholder_time_flattened()[source]
get_placeholder_flattened(keep_dims=False)[source]
Parameters:keep_dims (bool) – if set, it will add broadcast dimensions after the flattening behind the first axis
Return type:tf.Tensor
Returns:placeholder where all dynamic axes are flattened into a single axis. e.g. for the usual case (batch, time, dim), it becomes (batch’|time’, dim), or (batch, time, height, dim) will also become (batch’|time’, dim). with keep_dims, (batch, time, height, dim) will become (batch’|time’, 1, 1, dim).
feature_dim_axis[source]
get_axes(exclude_time=False, exclude_batch=False)[source]
Parameters:
  • exclude_time (bool) – will filter out the time-axis
  • exclude_batch (bool) – will filter out the batch-axis
Returns:

list of axes, like range(len(self.shape)), calculated with batch dim.

Return type:

list[int]

get_axes_from_description(axes)[source]
Parameters:axes (int|list[int]|str|list[str]|None) – one axis or multiple axis, or none. This is counted with batch-dim, which by default is axis 0 (see enforce_batch_dim_axis). It also accepts the special tokens “B”|”batch”, “spatial”, “spatial_except_time”, or “F”|”feature”, and more (see the code).
Returns:list of axes, counted with batch-dim
Return type:list[int]
get_axis_from_description(axis)[source]
Parameters:axis (int|str) –
Returns:axis, counted with batch-dim
Return type:int
get_batch_axis_excluding_batch(axis)[source]
Parameters:axis (int) – counted with batch-dim
Returns:axis counted without batch-dim
Return type:int
get_batch_axis(axis)[source]
Parameters:axis (int) – counted without batch-dim
Returns:axis counted with batch-dim
Return type:int
have_batch_axis()[source]
have_time_axis()[source]
have_feature_axis()[source]
is_time_axis_dynamic()[source]
Returns:whether there are different seq-lens for the time, or all the same (static)
Return type:bool
get_sequence_lengths()[source]
Returns:seq lens tensor of shape (batch,) of dtype int32
Return type:tf.Tensor
get_sequence_mask()[source]
Returns:seq mask of shape (batch,time) if we are batch-major, else (time,batch) if we are time-major
Return type:tf.Tensor
get_sequence_mask_broadcast()[source]
Returns:seq mask of shape ((batch,time) or (time,batch)) + (1,)s for remaining dims
Return type:tf.Tensor
get_batch_dim()[source]
Return type:tf.Tensor
get_spatial_batch_axes()[source]
Return type:list[int]
Returns:list of axes which are not feature and batch axes, counted with batch-dim.
get_spatial_axes()[source]
Return type:list[int]
Returns:list of axes which are not feature and batch axes, counted without batch-dim.
get_feature_batch_axes()[source]
Return type:list[int]
Returns:list of axes which are feature axes, counted with batch-dim. currently there is only one or zero such axis.
get_feature_axes()[source]
Return type:list[int]
Returns:list of axes which are feature axes, counted without batch-dim.
SpecialAxesNames = ('batch_dim_axis', 'time_dim_axis', 'feature_dim_axis')[source]
get_special_axes_dict(counted_with_batch_dim=True, include_batch_dim_axis=False, only_available=False)[source]
Parameters:
  • counted_with_batch_dim (bool) –
  • include_batch_dim_axis (bool) –
  • only_available (bool) –
Returns:

dict axis-name -> axis

Return type:

dict[str,int]

get_bc_spatial_batch_shape()[source]
Returns:shape which will broadcast along all spatial dimensions and time/batch dim
Return type:tuple[int|None]
get_bc_shape(opts=None)[source]
Parameters:opts (dict[str|list|tuple,int|str|None]|None) – key specifies the axes. value 1 (‘x’) is broadcasting, -1 (None) is not broadcasting Axes should not be defined multiple times. The default behavior if an axis is not specified is like get_bc_spatial_batch_shape(), i.e. it will broadcast in batch and spatial dims only.
Returns:shape where 1 means broadcasting, None or >1 means not broadcasting. can be used for TFUtil.dropout()
Return type:tuple[int|None]
TFUtil.init_horovod()[source]

Initializes Horovod. Provide this here such that we can remember whether we already initialized before.

class TFUtil.CustomUpdate[source]
set_on_var(var)[source]
Parameters:var (tf.Variable) – variable to update. this will be recognized by TFUpdater.Updater
update_var(var)[source]
Parameters:var (tf.Variable) – variable to update
Returns:operation which updates the variable, e.g. tf.assign_add(var, something)
Return type:tf.Operation
class TFUtil.CustomUpdateExpAverage(average, alpha)[source]

exponential moving average

Parameters:
  • average (tf.Tensor) –
  • alpha (float) –
update_var(var)[source]
Parameters:var (tf.Variable) – variable to update
Returns:operation which updates the variable, e.g. tf.assign_add(var, something)
Return type:tf.Operation
TFUtil.set_param_axes_split_info(param, axes_split_info)[source]
Parameters:
  • param (tf.Variable|tf.Tensor) –
  • axes_split_info (list[list[int]]) – e.g. [[n],[n]*4] for LSTM matrices
TFUtil.check_param_axes_split_info(param_shape, axes_split_info)[source]
Parameters:
  • param_shape (list[int]|tuple[int]) –
  • axes_split_info (list[list[int]]) – e.g. [[n],[n]*4] for LSTM matrices
TFUtil.get_param_axes_split_info(param)[source]

See set_param_axes_split_info().

Parameters:param (tf.Variable|tf.Tensor) –
Return type:list[list[int]]|None
TFUtil.transform_param_axes_split_info_to_new_shape(axes_split_info, new_shape)[source]

new_shape can be bigger or smaller than the old shape. In some simple cases, it is obvious how that should be done, e.g. [[a],[b]*4], [a*2,b*8] -> [[a*2],[b*2]*4] In some, it is not so. E.g. [[a+b],[b]*4], [a+b*2,b*8] -> [[a+b*2],[b*2]*4]. See test cases as well, test_transform_param_axes_split_info_to_new_shape(). No TF involved here, however, fits better to the functions above.

Parameters:
  • axes_split_info (list[list[int]]) –
  • new_shape (list[int]|tuple[int]) –
Returns:

new axes-split-info for the new shape

Return type:

list[list[int]]

TFUtil.copy_with_new_split_axes(old_axis_splits, new_axis_splits, old_values, new_values=None)[source]

On Numpy arrays only, however, fits better to the functions above.

Parameters:
  • old_axis_splits (list[list[int]]) –
  • new_axis_splits (list[list[int]]) –
  • old_values (numpy.ndarray) –
  • new_values (numpy.ndarray) –
Returns:

new values

Return type:

numpy.ndarray

class TFUtil.OutputWithActivation(x, act_func=None)[source]
Parameters:
  • x (tf.Tensor) –
  • act_func (None|(tf.Tensor)->tf.Tensor) –
is_softmax_act_func()[source]
get_logits()[source]
Return type:tf.Tensor
Returns:logits. logits are (not necessarily normalized) log probabilities, i.e. the input of softmax.

This call assumes that self.y is in probability space.

get_log_output()[source]
Return type:tf.Tensor
Returns:tf.log(output)
TFUtil.variable_scalar_summaries_dict(x, name=None)[source]

Collects all interesting information about x, such as min/max/mean, etc. (all scalars). This is used by variable_summaries().

Parameters:
  • x (tf.Tensor|tf.Variable) –
  • name (str) –
Returns:

dicth with key -> scalar info, e.g. with “%s_mean” % name -> tf.reduce_mean(x)

Return type:

dict[str,tf.Tensor]

TFUtil.variable_summaries(var, name=None, with_histogram=False)[source]

Attach a lot of summaries to a Tensor (for TensorBoard visualization). Also see variable_scalar_summaries_dict().

Parameters:
  • var (tf.Tensor|tf.Variable) –
  • name (str) –
  • with_histogram (bool) – adds histogram. note that this can add noticeable overhead
Returns:

nothing, use tf.summary.merge_all() to collect the summaries

TFUtil.get_current_var_scope_name()[source]
Returns:current absolute variable scope name, via tf.variable_scope
Return type:str
TFUtil.get_current_name_scope()[source]
Returns:current absolute name scope, via tf.name_scope
Return type:str

http://stackoverflow.com/questions/40907769/how-to-get-current-tensorflow-name-scope

Note that this is a private member and might break at some point. Note also that this does not need to be the same as get_current_var_scope_name().

TFUtil.reuse_name_scope(name, absolute=None, **kwargs)[source]

Context manager to reuse an already created scope. We try to both set the variable scope and the name scope.

Parameters:
  • name (str|tf.VariableScope) – relative or absolute name scope (absolute if absolute=True or if tf.VariableScope). must not end with “/”.
  • absolute (bool) – if True it will be absolute
  • kwargs – passed on to tf.variable_scope
Returns:

yields the variable_scope

TFUtil.opt_reuse_name_scope(name)[source]
Parameters:name (str|tf.VariableScope) –
Returns:yields the variable_scope
TFUtil.get_name_scope_of_tensor(x)[source]
Parameters:x (tf.Tensor) – has name e.g. “layer0/rec/W:0”
Returns:the name scope of x, e.g. “layer0/rec”
Return type:str
TFUtil.get_base_name(x)[source]
Parameters:x (tf.Tensor) – has name e.g. “layer0/rec/W:0”
Returns:return the base name, e.g. “W”, without the output index
TFUtil.reuse_name_scope_of_tensor(x, prefix='', postfix='')[source]
Parameters:
  • x (tf.Tensor) – has name e.g. “layer0/rec/W:0”
  • prefix (str) –
  • postfix (str) –
Returns:

reuse the name scope of x, e.g. “layer0/rec”, yields scope

TFUtil.var_creation_scope()[source]
If you create a variable inside of a while-loop, you might get the following error:
InvalidArgumentError: The node ‘while/w/Assign’ has inputs from different frames. The input ‘while/j’ is in frame ‘while/while/’. The input ‘while/w’ is in frame ‘’.

Also see tests/test_TFUtil.py:test_loop_var_creation(). Related TF bugs:

The solution is to reset the current frame. Resetting all control dependencies has this effect. See also same_context()

class TFUtil.FlipGradientBuilder[source]

Gradient Reversal Layer. Discussion:

Code from here:
https://github.com/pumpikano/tf-dann/blob/master/flip_gradient.py

Also see CustomGradient which is more generic.

TFUtil.lookup_grad_func_by_name(op_type)[source]
Parameters:op_type (str) –
Returns:function grad_func(op, grad), or raises LookupError
TFUtil.opt_register_grad_func(op_type, grad_func, assert_is_same=True)[source]
Parameters:
  • op_type (str) –
  • grad_func – function grad_func(op, grad)
  • assert_is_same (bool) –
TFUtil.identity_with_check_numerics(x, with_grad=True, name='identity_with_check_numerics')[source]

Returns identity(x), but with additional check_numerics control dependency, and optionally the same for its gradient. See also TFUpdater.add_check_numerics_ops(), which will add checks for the whole graph.

Parameters:
  • x (tf.Tensor) –
  • with_grad (bool) – whether the check will also be added for the gradient
  • name (str) –
Return type:

tf.Tensor

TFUtil.check_input_ndim(x, ndim)[source]
Parameters:
  • x (tf.Tensor) –
  • ndim (int) –
Returns:

x with check added

Return type:

tf.Tensor

TFUtil.check_input_ndim_equal_offset(x, y, y_ndim_offset=0)[source]
Parameters:
  • x (tf.Tensor) –
  • y (tf.Tensor) –
  • y_ndim_offset (int) –
Returns:

x with check added such that ndim(x) == ndim(y) + y_ndim_offset

Return type:

tf.Tensor

TFUtil.check_input_dim(x, axis, dim)[source]
Parameters:
  • x (tf.Tensor) –
  • axis (int) – which axis to check
  • dim (int|tf.Tensor) –
Returns:

x with check added

Return type:

tf.Tensor

TFUtil.check_dim_equal(x, x_axis, y, y_axis, extra_msg=())[source]
Parameters:
  • x (tf.Tensor) –
  • x_axis (int) – which axis to check
  • y (tf.Tensor) –
  • y_axis (int) – which axis to check
  • extra_msg (list[str]|tuple[str]) – will be printed additionally if it fails
Returns:

x with check added that shape(x)[x_axis] == shape(y)[y_axis]

Return type:

tf.Tensor

TFUtil.check_shape_equal(x, y)[source]
Parameters:
  • x (tf.Tensor) –
  • y (tf.Tensor) –
Returns:

x with check added that shape(x) == shape(y)

Return type:

tf.Tensor

TFUtil.get_shape_dim(x, axis, name='shape_dim')[source]
Parameters:
  • x (tf.Tensor) –
  • axis (int) – which axis
  • name (str) –
Returns:

x.shape[axis] either as a static int or otherwise as an expression

Return type:

int|tf.Tensor

TFUtil.get_shape(x)[source]
Parameters:x (tf.Tensor) –
Returns:list of scalars, which are either int if known statically, or otherwise expressions
Return type:list[int|tf.Tensor]
TFUtil.get_ndim(x)[source]
Parameters:x (tf.Tensor) –
Returns:x.ndim either as a static int or otherwise as an expression
Return type:int|tf.Tensor
TFUtil.get_range(start, stop=<class 'Util.NotSpecified'>)[source]
Parameters:
  • start (int|tf.Tensor|None) –
  • stop (int|tf.Tensor|None) –
Returns:

either tuple(range(start, stop)) or the same as a symbolic expression

Return type:

tuple[int]|tf.Tensor

TFUtil.identity_with_ops(x, ops)[source]
Parameters:
  • x (tf.Tensor) –
  • -> list[tf.Operation|tf.Tensor] ops (()) –
Returns:

x with all ops executed

Return type:

tf.Tensor

TFUtil.setup_tf_thread_pools(num_threads=None, log_file=None, tf_session_opts=None)[source]

See here for documentation of intra_op_parallelism_threads and inter_op_parallelism_threads: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/protobuf/config.proto

intra_op_parallelism_threads is used for the LocalDevice::EigenThreadPoolInfo, which is always global. https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/common_runtime/local_device.cc

inter_op_parallelism_threads is used for the (global if not use_per_session_threads) session thread pool. https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/common_runtime/direct_session.cc

TF will setup the thread pools on first usage. That can happen quite early, esp for intra_op_parallelism_threads. E.g. list_local_devices() will trigger this, i.e. any call to is_gpu_available() or print_available_devices(). For debugging, you can set the env-var TF_CPP_MIN_VLOG_LEVEL=1 and then check for these message:

Local device intra op parallelism threads: 4
Direct session inter op parallelism threads: 4

Thus, call this function as early as possible with your preferred number of threads, used for both thread pools. It will create a dummy session and directly close it again, but if you use the global thread pools, those settings will remain for further sessions. This function will only execute on the first call.

Parameters:
  • num_threads (int) – used for both intra and inter parallelism thread pools
  • log_file (stream|None) –
  • tf_session_opts (dict[str]) –
TFUtil.check_initial_tf_thread_pool_init(tf_session_opts=None)[source]
TFUtil.get_tf_list_local_devices(tf_session_opts=None)[source]

This uses tensorflow.device_lib.list_local_devices(). Note that a call to this will trigger the internal TF thread pool inits, so you should call setup_tf_thread_pools() first. Note that this will list all available devices. Any TF session might only use a subset of these. You can get the list available in a given TF session by tf.Session.list_devices().

Parameters:tf_session_opts (dict[str]|None) – if given, will init a temp tf.Session with these opts
Return type:list[tensorflow.core.framework.device_attributes_pb2.DeviceAttributes|_DeviceAttributes]
TFUtil.print_available_devices(tf_session_opts=None, file=None)[source]

Prints the available TF devices on file (stdout by default). This uses tensorflow.device_lib.list_local_devices(). Note that a call to this will trigger the internal TF thread pool inits, so you should call setup_tf_thread_pools() first.

Parameters:
  • tf_session_opts (dict[str]|None) – if given, will init a temp tf.Session with these opts
  • file (io.FileIO) –
TFUtil.is_gpu_available()[source]

Returns whether TensorFlow can access a GPU. This uses tensorflow.device_lib.list_local_devices(). Note that a call to this will trigger the internal TF thread pool inits, so you should call setup_tf_thread_pools() first.

Return type:bool
TFUtil.get_available_gpu_devices()[source]

Returns a list of available GPU devices. This uses tensorflow.device_lib.list_local_devices(). Note that a call to this will trigger the internal TF thread pool inits, so you should call setup_tf_thread_pools() first.

Return type:list[tensorflow.core.framework.device_attributes_pb2.DeviceAttributes]
TFUtil.get_available_gpu_min_compute_capability()[source]

Uses get_available_gpu_devices().

Returns:e.g. 3.0, or 5.0, etc, or None
Return type:float|None
TFUtil.dot(a, b)[source]
Parameters:
  • a (tf.Tensor) – shape […da…,d]
  • b (tf.Tensor) – shape [d,…db…]
Returns:

tensor of shape […da…,…db…]

Return type:

tf.Tensor

TFUtil.identity(x)[source]
Parameters:x (tf.Tensor) –
Return type:tf.Tensor
TFUtil.get_activation_function(s)[source]
Parameters:s (str|None) –
Return type:(tf.Tensor) -> tf.Tensor
TFUtil.random_uniform_abs_initializer(limit, **kwargs)[source]
TFUtil.xavier_initializer(uniform=True, seed=None, dtype=tf.float32)[source]

Alias for tf.glorot_uniform_initializer or tf.glorot_normal_initializer.

Parameters:
  • uniform (bool) – uniform or normal distribution
  • seed (int) –
  • dtype (tf.DType) –
Returns:

((tuple[int]) -> tf.Tensor) | tf.Initializer

TFUtil.wrap_distribution_non_zero(x, zero_limit, limit)[source]
Parameters:
  • x (tf.Tensor) – values in [-limit,limit]
  • zero_limit (float) –
  • limit (float) –
Returns:

same shape as x. rescale and shifts such that values from [-zero_limit,zero_limit] are excluded. still values are in [-limit,limit].

Return type:

tf.Tensor

class TFUtil.VarianceScalingNonZero(non_zero_fraction=0.5, **kwargs)[source]

Same as tf.VarianceScaling, i.e. truncated normal or uniform from [-limit,limit] for some limit, except that we exclude the range [-limit*non_zero_fraction,limit*non_zero_fraction]. non_zero_fraction=0 would yield no difference.

For reference, to get the behavior of glorot_uniform, use these args:
mode=”fan_avg”, distribution=”uniform”
TFUtil.variance_scaling_non_zero_initializer[source]

alias of TFUtil.VarianceScalingNonZero

TFUtil.load_txt_file_initializer(filename, dtype=tf.float32)[source]
Parameters:
  • filename (str) –
  • dtype (tf.DType) –
Returns:

function, when called, will return the content

Return type:

()->tf.Tensor

TFUtil.get_initializer(s, seed=None, eval_local_ns=None, dtype=tf.float32)[source]
Parameters:
  • s (str|dict[str]|float) – e.g. “glorot_uniform” or “truncated_normal” or “orthogonal”, or config dict with “class”, or string to be `eval`ed if it contains “(“. constant if a float is given.
  • seed (int|tf.Tensor) –
  • eval_local_ns (dict[str]|None) –
  • dtype (tf.DType|str) –
Returns:

(function (shape) -> tf.Tensor) | tf.Initializer

Return type:

((tuple[int]) -> tf.Tensor) | tf.Initializer

TFUtil.dropout(x, keep_prob, noise_shape=None, seed=None, name=None, cond_on_train=False)[source]

Computes dropout. Like tf.nn.dropout() but avoid tf.div() if possible.

Parameters:
  • x (tf.Tensor) –
  • keep_prop (float|tf.Tensor) –
  • noise_shape (tf.Tensor|tuple[int|None]) – 1 will broadcast in that dimension, None will not broadcast
  • seed (int) –
  • name (str) –
  • cond_on_train (bool) – automatically wrap through cond_on_train_flag()
TFUtil.layer_norm(x, gain, bias, axis, epsilon=1e-06)[source]

Layer normalization. Also see openai_layer_norm(). Also see tensorflow.contrib.layers.layer_norm().

Parameters:
  • x (tf.Tensor) –
  • gain (tf.Tensor) –
  • bias (tf.Tensor) –
  • axis (int) –
  • epsilon (float) – OpenAI uses 1e-6, TF contrib uses 1e-12, pbhatia243 uses 1e-5.
Return type:

tf.Tensor

TFUtil.openai_layer_norm(x, gain, bias, axis, epsilon=1e-06)[source]

Layer normalization, like layer_norm(), but fast kernel by OpenAI (implemented as part of their blocksparse). To use it, init the git submodule in extern/blocksparse.

Parameters:
  • x (tf.Tensor) –
  • gain (tf.Tensor) –
  • bias (tf.Tensor) –
  • axis (int) –
  • epsilon (float) –
Return type:

tf.Tensor

TFUtil.swapaxes(x, axis1, axis2)[source]

Also see move_axis() or dimshuffle().

Parameters:
  • x (tf.Tensor) –
  • axis1 (tf.Tensor|int) –
  • axis2 (tf.Tensor|int) –
Returns:

tensor with swapped axes, like numpy.swapaxes

Return type:

tf.Tensor

TFUtil.move_axis(x, old_axis, new_axis, name='move_axis')[source]

Also see swapaxes() or dimshuffle().

Parameters:
  • x (tf.Tensor) –
  • old_axis (int) – can also be negative
  • new_axis (int) – can also be negative
  • name (str) – name of the scope
TFUtil.sequence_mask(lengths, **kwargs)[source]

Wraps around tf.sequence_mask(). It will cache the value inside the passed object so that we don’t recompute it multiple times.

Parameters:
  • lengths (tf.Tensor) – shape (batch,)
  • kwargs – passed on to tf.sequence_mask
Returns:

tensor mask of shape (batch,maxlen/time). default dtype is bool unless you specify something else

Return type:

tf.Tensor

TFUtil.sequence_mask_time_major(lengths, **kwargs)[source]

Wraps around tf.transpose(tf.sequence_mask(), (1,0)). It will cache the value inside the passed object so that we don’t recompute it multiple times.

Parameters:
  • lengths (tf.Tensor) – shape (batch,)
  • kwargs – passed on to tf.sequence_mask
Returns:

mask of shape (maxlen/time,batch)

Return type:

tf.Tensor

TFUtil.directed(x, direction)[source]

If direction == 1 or direction is None, returns just x. If direction == -1, returns reversed(x).

Parameters:
  • x (tf.Tensor) –
  • direction (int|None) – -1 or 1 (or None)
Return type:

tf.Tensor

TFUtil.reversed(x)[source]

Just returns x[::-1]. It will cache the value inside the passed object so that we don’t recompute it multiple times.

Parameters:x (tf.Tensor) –
Return type:tf.Tensor
TFUtil.flatten_with_seq_len_mask(x, seq_lens, time_major=False)[source]
Parameters:
  • x (tf.Tensor) – shape (batch,time,…s…) with time_major=False or otherwise shape (time,batch,…s….)
  • seq_lens (tf.Tensor) – shape (batch,) of int32
  • time_major (bool) – if the time-dim is the first dimension in x
Returns:

tensor of shape (time’, …s…) where time’ = sum(seq_len) <= batch*time

Return type:

tf.Tensor

TFUtil.expand_dims_unbroadcast(x, axis, dim, name='expand_dims_unbroadcast')[source]
Parameters:
  • x (tf.Tensor|float|int) –
  • axis (int|tf.Tensor) – new axis
  • dim (int|tf.Tensor) – dimension for axis
  • name (str) – scope name
Returns:

if x is of shape (a,b,c) and axis=0, then we return (dim,a,b,c)

Return type:

tf.Tensor

TFUtil.expand_multiple_dims(x, axes, name='expand_multiple_dims')[source]
Parameters:
  • x (tf.Tensor) –
  • axes (list[int]|tuple[int]) – after completion, tf.shape(y)[axis] == 1 for axis in axes
  • name (str) – scope name
Returns:

y where we have a new broadcast axis for each axis in axes

Return type:

tf.Tensor

TFUtil.tile_transposed(x, axis, multiples)[source]

Example: x with shape (D,), tf.tile(x, [N]) can be reshaped into (N,D), while tile_transposed(x, axis=0, multiples=N) can be reshaped into (D,N).

Parameters:
  • x (tf.Tensor) –
  • axis (int) –
  • multiples (int|tf.Tensor) –
Returns:

tensor with shape[axis] == x.shape[axis] * multiples

Return type:

tf.Tensor

TFUtil.constant_with_shape(x, shape, dtype=None, name='constant_with_shape')[source]
Parameters:
  • x (tf.Tensor|float|int|bool) – scalar
  • shape (list[tf.Tensor|int]|tuple[tf.Tensor|int]|tf.Tensor) –
  • dtype (tf.DType) –
  • name (str) –
Returns:

x of the specified shape

Return type:

tf.Tensor

TFUtil.dimshuffle(x, axes, name='dimshuffle')[source]

Like Theanos dimshuffle. Combines tf.transpose, tf.expand_dims and tf.squeeze.

Parameters:
  • x (tf.Tensor) –
  • axes (list[int|str]|tuple[int|str]) –
  • name (str) – scope name
Return type:

tf.Tensor

TFUtil.sparse_labels_with_seq_lens(x, seq_lens, dtype=tf.int32, collapse_repeated=False)[source]
Parameters:
  • x (tf.Tensor) – shape (batch,time) -> index, some int type
  • seq_lens (tf.Tensor|None) – shape (batch,) of int32|int64
  • dtype (tf.DType|None) – if given, will cast the x values to this type. ctc_loss() wants int32
  • collapse_repeated (bool) – like uniq() behavior
Returns:

SparseTensor, e.g. input for tf.nn.ctc_loss(), and seq_lens of shape (batch,)

Return type:

(tf.SparseTensor, tf.Tensor)

TFUtil.sparse_labels(x, seq_lens, dtype=tf.int32, collapse_repeated=False)[source]
Parameters:
  • x (tf.Tensor) – shape (batch,time) -> index, some int type
  • seq_lens (tf.Tensor|None) – shape (batch,) of int32|int64
  • dtype (tf.DType|None) – if given, will cast the x values to this type. ctc_loss() wants int32
  • collapse_repeated (bool) – like uniq() behavior
Returns:

SparseTensor, e.g. input for tf.nn.ctc_loss()

Return type:

tf.SparseTensor

TFUtil.uniq(x)[source]
Parameters:x (tf.Tensor) – 1D shape (time,) -> index, some int type
Returns:like numpy.uniq. unlike tf.unique which will never repeat entries.

Example: uniq([0, 0, 1, 1, 0, 0]) == [0, 1, 0], tf.unique([0, 0, 1, 1, 0, 0]) == [0, 1]. For a batched variant, see batched_uniq, or sparse_labels() with option collapse_repeated.

TFUtil.batched_uniq(x, seq_lens)[source]
Parameters:
  • x (tf.Tensor) – shape (batch,time) -> index, some int type
  • seq_lens (tf.Tensor|None) – shape (batch,) of int32|int64
Returns:

tuple (z, new_seq_lens), where z is of shape (batch, max_new_time), max_new_time = max(new_seq_lens), seq_lens is of shape (batch,).

Return type:

(tf.Tensor, tf.Tensor)

TFUtil.matrix_triangular(shape, dtype=tf.float32, lower=False, upper=False)[source]
Parameters:
  • shape (tuple[int|tf.Tensor]|tf.Tensor) –
  • dtype (tf.DType) –
  • lower (bool) –
  • upper (bool) –
Return type:

tf.Tensor

class TFUtil.VariableAssigner(var)[source]
Parameters:var (tf.Variable) –
assign(value, session)[source]
Parameters:
  • value (numpy.ndarray|int|float|list[str]) –
  • session (tf.Session) –
class TFUtil.CudaEnv[source]
verbose_find_cuda = False[source]
is_available()[source]
get_compiler_opts()[source]
get_compiler_bin()[source]
classmethod get_instance()[source]
Return type:CudaEnv
class TFUtil.OpCodeCompiler(use_cuda_if_available=True, cuda_auto_min_compute_capability=True, include_paths=(), ld_flags=(), **kwargs)[source]

Helper class to compile TF ops on-the-fly, similar as Theano. https://www.tensorflow.org/versions/master/how_tos/adding_an_op/ https://github.com/tensorflow/tensorflow/blob/master/tensorflow/docs_src/extend/adding_an_op.md

CacheDirName = 'returnn_tf_cache/ops'[source]
load_tf_module()[source]
class TFUtil.TFNativeUtilCompiler(include_paths=(), ld_flags=(), **kwargs)[source]

Helper class to compile TF utility functions on-the-fly.

CacheDirName = 'returnn_tf_cache/tf_utils'[source]
TFUtil.make_var_tuple(v)[source]
Parameters:v (tf.Tensor|list[tf.Tensor]|tuple[tf.Tensor]) –
Returns:tuple of tensors
Return type:tuple[tf.Tensor]
TFUtil.add_scaled_noise_to_gradients(grads_and_vars, gradient_noise_scale, sparse_grads=False)[source]

Adds scaled noise from a 0-mean normal distribution to gradients. Adapted from tf.contrib.layers.optimizers.

Parameters:
  • tf.Variable)] grads_and_vars (list[(tf.Tensor|tf.IndexedSlices,) –
  • gradient_noise_scale (float) – used as stddev for tf.truncated_normal().
  • sparse_grads (bool) – for sparse gradients (tf.IndexedSlices), it will only add the noise to the indexed values. Seems broken in some cases? Needs debugging.
Returns:

adapted grads_and_vars

Return type:

list[(tf.Tensor|tf.IndexedSlices, tf.Variable)]

class TFUtil.CustomGradient[source]

Utility functions to specify a custom gradient for a given function, which will be wrapped around via TF Defun().

Also see FlipGradientBuilder.

Defun(*input_types, **kwargs)[source]
Parameters:
  • tf.Tensor) -> tf.Tensor grad_op ((tf.Operation,) –
  • input_types (list[tf.DType]) –
  • kwargs (dict[str]) – passed to self.register()
Returns:

function decorator

Return type:

((tf.Tensor) -> tf.Tensor) -> ((tf.Tensor) -> tf.Tensor)

register(input_types, op, grad_op, name=None)[source]
Parameters:
  • input_types (list[tf.DType]) –
  • -> tf.Tensor op ((tf.Tensor)) –
  • tf.Tensor) -> tf.Tensor grad_op ((tf.Operation,) – args are (op, out_grad) and it must return in_grad
  • name (str) – optional func_name
Returns:

op

Return type:

(tf.Tensor) -> tf.Tensor

register_generic_loss_and_error_signal()[source]

If you want to use generic_loss_and_error_signal() at some point, call this as early as possible, because of https://github.com/tensorflow/tensorflow/issues/6804.

generic_loss_and_error_signal(loss, x, grad_x)[source]

Wrapper around self.register(). Expects that loss = loss(x), and grad_x = partial loss / partial x.

Parameters:
  • loss (tf.Tensor) –
  • x (tf.Tensor) –
  • grad_x (tf.Tensor) –
Returns:

loss but with the gradient for x

Return type:

tf.Tensor

class TFUtil.SyntheticGradient[source]
class Scope[source]
register_loss(loss)[source]
Parameters:loss (tf.Tensor) –
exit()[source]
as_fetch_dict()[source]
class ScopeCtxThreadLocal[source]
scope = None[source]
scope_ctx = <TFUtil.SyntheticGradient.ScopeCtxThreadLocal object>[source]
classmethod enter_gradient_scope()[source]
Return type:SyntheticGradient.Scope
classmethod exit_gradient_scope()[source]
classmethod gradient_scope()[source]

When gradients will be calculated (e.g. via some optimizer.minimize()), the synthetic gradient needs to collect the gradient prediction losses. This is done via this global scope.

classmethod synthetic_gradient(x, synthetic_grad_x)[source]
Parameters:
  • x (tf.Tensor) –
  • synthetic_grad_x (tf.Tensor) –
Returns:

x, where the gradient is overwritten by synthetic_grad_x, and when calculated, the gradient prediction loss will be added to cls.scope.

Return type:

tf.Tensor

TFUtil.filter_grad(x, threshold, axis)[source]
Parameters:
  • x (tf.Tensor) –
  • threshold (float) – all grads going through x which max(grad**2) is over the threshold are removed
  • axis (int|list[int]) – max(grad**2) will be reduced over this axis
Returns:

identity(x) with custom gradient

Return type:

tf.Tensor

TFUtil.debugRegisterBetterRepr()[source]

Some types don’t have good __repr__ implementations by default (for the current TF version). For debugging, it can be helpful to give some more info. This monkey-patches clazz.__repr__ of some TF classes if they are object.__repr__.

TFUtil.cond(pred, fn1, fn2, name=None)[source]

This is a wrapper around tf.control_flow_ops.cond(). This will be a branched execution, i.e. either fn1() or fn2() will be executed, or at least the resulting graph will be evaluated. If pred can is constant at the call, only the corresponding fn will be called. This is similar as the TF internal _smart_cond(). And similar as tf.contrib.framework.smart_cond.

Parameters:
  • pred (tf.Tensor|bool) –
  • fn1 (()->(tf.Tensor|list[tf.Tensor])) –
  • fn2 (()->(tf.Tensor|list[tf.Tensor])) –
  • name (str) –
Returns:

fn1() if pred else fn2()

Return type:

tf.Tensor|list[tf.Tensor]

TFUtil.single_strided_slice(x, axis, begin=None, end=None, step=None)[source]
Parameters:
  • x (tf.Tensor) –
  • axis (int|tf.Tensor) –
  • begin (int|tf.Tensor|None) –
  • end (int|tf.Tensor|None) –
  • step (int|tf.Tensor|None) –
Returns:

e.g. if axis == 0, returns x[begin:end:step], if axis == 1, returns x[:, begin:end:step], etc.

Return type:

tf.Tensor

TFUtil.circular_pad(x, paddings, axes=None)[source]
Parameters:
  • x (tf.Tensor) – shape (…, height, width)
  • (int,int))|tf.Tensor paddings (int|((int,int),) – how much to add ((top,bottom),(left,right))
Returns:

tensor with shape (…, top + height + bottom, left + width + right)

Return type:

tf.Tensor

TFUtil.spatial_smoothing_energy(x, dim, use_circular_conv=True)[source]
Parameters:
  • x (tf.Tensor) – shape (…, dim)
  • dim (int) – last dimension of x
  • use_circular_conv (bool) – whether to use circular convolution, via circular_pad
Return type:

tf.Tensor

Returns:

energy of shape (…)

Via Achieving Human Parity in Conversational Speech Recognition, Microsoft, 2017. Interpret the last dimension as 2D (w, h) and apply some high-pass filter on it.

TFUtil.nan_to_num(x, nan_num=0, inf_num=1e+30)[source]

Like numpy.nan_to_num().

Parameters:
  • x (tf.Tensor|tf.IndexedSlices) –
  • nan_num (float|tf.Tensor) –
  • inf_num (float|tf.Tensor) –
Returns:

x with replaced nan and inf

TFUtil.identity_op_nested(x, name='identity')[source]
Parameters:
  • x (tf.Tensor|list[tf.Tensor]|dict[str,tf.Tensor]) –
  • name (str) –

:rtype tf.Tensor|list[tf.Tensor]|dict[str,tf.Tensor]

TFUtil.nd_indices(indices, batch_axis=0)[source]
Parameters:indices (tf.Tensor) – e.g. (batch, …) -> index
Returns:extended indices with batch-idx which can be used for tf.gather_nd, i.e. in the example of shape (batch, …, 2) where the 2-tuple represents (batch_idx, index).
Return type:tf.Tensor
TFUtil.stop_event_writer_thread(event_writer)[source]

There is a bug in TensorFlow (at least 1.1.0) (https://github.com/tensorflow/tensorflow/issues/4820) that the event writer thread is never stopped. This will try to stop it. Only do it if you don’t use the event writer anymore.

Parameters:event_writer (tensorflow.python.summary.writer.event_file_writer.EventFileWriter) –
TFUtil.optional_add(*args)[source]
Parameters:args (list[tf.Tensor|None]|tf.Tensor) –
Return type:tf.Tensor|None
Returns:sums all non-None values, or returns None if there are none
TFUtil.opt_logical_and(*args)[source]
Parameters:args (list[tf.Tensor|bool]) –
Returns:basically logical_and(*args), but leaves out all constants
Return type:tf.Tensor|bool
TFUtil.windowed_nd(source, window_size, window_left=None, window_right=None, padding='same', time_axis=1, new_window_axis=2)[source]

Constructs a new “window” axis which is a moving input over the time-axis. If you want to take out a window, i.e. a slice, see slice_nd().

Parameters:
  • source (tf.Tensor) – N-D tensor of shape (…, n_time, …)
  • window_size (int|tf.Tensor) – window size
  • window_left (int|tf.Tensor|None) –
  • window_right (int|tf.Tensor|None) –
  • padding (str) – “same” or “valid”
  • time_axis (int) –
  • new_window_axis (int) –
Returns:

tensor of shape (…, n_time, …, window, …)

Return type:

tf.Tensor

TFUtil.slice_nd(x, start, size, seq_lens=None)[source]
Parameters:
  • x (tf.Tensor) – shape (B, T, …)
  • start (tf.Tensor) – shape (B,), int32
  • size (int) –
  • seq_lens (tf.Tensor|None) – shape (B,), int32, <= T. if None, [T]*B is assumed.
Returns:

[x[start_1:size], x[start_2:size], …, x[start_B:size]], shape (B, size, …) Like slice_pad_zeros(), the size in the first axis will always be size, and we will pad with zeros.

Return type:

tf.Tensor

TFUtil.global_tensor(f, name)[source]

This creates a global accessible tensor in the graph to be reused later, i.e. on the second call given a unique name, it will not create a new tensor but return the previously created tensor. This is for the current graph, i.e. if there is a new graph, it will recreate the tensor.

Parameters:
  • -> tf.Tensor f (()) – callable which creates the tensor
  • name (str) – global reference name for the tensor. should be a valid scope name
Returns:

the tensor

Return type:

tf.Tensor

TFUtil.get_global_train_flag_placeholder()[source]

Also consider TFNetwork.get_current_network().train_flag(), or get_global_train_flag().

Returns:bool scalar tensor
Return type:tf.Tensor
TFUtil.get_global_train_flag()[source]
Return type:tf.Tensor|bool
Returns:global train flag
TFUtil.cond_on_train_flag(fn_train, fn_eval)[source]

Uses fn_train() or fn_eval() base on train_flag. It will be a branched evaluation. train_flag is determined via get_global_train_flag().

Parameters:
  • fn_train (()->tf.Tensor) –
  • fn_eval (()->tf.Tensor) –
Returns:

fn_train() if self.train_flag else fn_eval()

Return type:

tf.Tensor

TFUtil.get_random_seed()[source]
Return type:int|None
TFUtil.encode_raw(x, axis=-1, seq_lens=None)[source]

The inverse function of tf.decode_raw(). Also see: https://stackoverflow.com/questions/43403147/how-to-create-a-encode-raw-tensorflow-function

Parameters:
  • x (tf.Tensor) – of integer types [0,255], will get casted to uint8
  • axis (int) – the axis to reduce-join the string. decode_raw has added it at the end
  • seq_lens (tf.Tensor|None) – must have same shape as x after reduce-joining. Note that using seq_lens will make our output not compatible with tf.decode_raw() anymore because tf.decode_raw() requires all strings to be of the same length.
Returns:

string tensor

Return type:

tf.Tensor

TFUtil.map_labels(x, label_map, name='map_labels')[source]
Parameters:
  • x (tf.Tensor|tf.SparseTensor) – values of integer types
  • label_map (dict[int,int|None]) – should be dense on input
  • name (str) –
Returns:

mapped values

Return type:

tf.Tensor

TFUtil.remove_labels(x, labels)[source]
Parameters:
  • x (tf.SparseTensor) – sequences, i.e. the indices are interpret as (batch,time)
  • labels (set[int]|list[int]) –
Returns:

x where all provided labels are removed, and the indices are changed accordingly

TFUtil.pad_zeros_in_axis(x, before=0, after=0, axis=0)[source]
Parameters:
  • x (tf.Tensor) –
  • before (int|tf.Tensor) –
  • after (int|tf.Tensor) –
  • axis (int) –
Returns:

TFUtil.slice_pad_zeros(x, begin, end, axis=0)[source]
Parameters:
  • x (tf.Tensor) – of shape (…, time, …)
  • begin (int|tf.Tensor) –
  • end (int|tf.Tensor) –
  • axis (int) –
Returns:

basically x[begin:end] (with axis==0) but if begin < 0 or end > x.shape[0], it will not discard these frames but pad zeros, such that the resulting shape[0] == end - begin.

Return type:

tf.Tensor

TFUtil.post_control_dependencies(x, updates)[source]
Parameters:
  • x (tf.Tensor|list[tf.Tensor]|dict[str,tf.Tensor]) –
  • updates (list[tf.Operation]) –
Returns:

identity(x) with control_dependencies(updates)

Return type:

tf.Tensor|list[tf.Tensor]|dict[str,tf.Tensor]

TFUtil.sequential_control_dependencies(l)[source]

tf.control_dependencies but each operation will be created such that it is executed after the ones coming before in the list, i.e. l[0] is executed first, l[-1] is executed last.

Parameters:l (list[()->(tf.Operation|tf.Tensor)]) –
TFUtil.global_queue(name, queue_type, capacity, dtypes, shapes=None, names=None)[source]
Parameters:
  • queue_type ((args)->tf.QueueBase) – some function which creates a queue
  • name (str) – global name
  • dtypes (list[tf.DType|str]) –
  • shapes (list[tf.TensorShape|tuple[int|None]]|None) –
  • names (list[str]|None) –
Return type:

tf.QueueBase

TFUtil.init_variable_if_needed(v)[source]
Parameters:v (tf.Variable) –
Return type:tf.Operation
TFUtil.auto_init_var(v)[source]
Parameters:v (tf.Variable) –
Returns:a reference to the var via tf.identity
Return type:tf.Tensor
TFUtil.true_once()[source]
Returns:tensor which will be True once and then always False Internally, this creates a non-trainable variable as a helper.
Return type:tf.Tensor
TFUtil.raise_OutOfRangeError()[source]
Returns:an op which raises an OutOfRangeError
Return type:tf.Operation
TFUtil.enforce_copy(x)[source]
Parameters:x (tf.Tensor|tf.Variable) –
Returns:copy of input, i.e. enforces that this is not a ref
TFUtil.view_as(x, dtype)[source]

Does the numpy.view equivalent. Note that the current implementation is inefficient (uses tf.py_func) and CPU-only. Also see tf.tf.bitcast.

Parameters:
  • x (tf.Tensor) –
  • dtype (tf.DType) –
Returns:

x.view(dtype) equivalent (see numpy.view)

TFUtil.broadcast_gradient_args(shape_x, shape_y)[source]
Parameters:
  • shape_x (tf.Tensor) –
  • shape_y (tf.Tensor) –
Returns:

(axis reduce arg for grad x, axis reduce arg for grad y)

Return type:

(tf.Tensor, tf.Tensor)

TFUtil.maximum_with_identity_grad(x, y)[source]
Parameters:
  • x (tf.Tensor) –
  • y (tf.Tensor) –
Returns:

tf.maximum(x, y) where each will receive the gradient

Return type:

tf.Tensor

TFUtil.minimum_with_identity_grad(x, y)[source]
Parameters:
  • x (tf.Tensor) –
  • y (tf.Tensor) –
Returns:

tf.maximum(x, y) where each will receive the gradient

Return type:

tf.Tensor

TFUtil.clip_by_value_with_identity_grad(x, clip_value_min, clip_value_max)[source]
Parameters:
  • x (tf.Tensor) –
  • clip_value_min (tf.Tensor|float) –
  • clip_value_max (tf.Tensor|float) –
Returns:

tf.clip_by_value(x, clip_value_min, clip_value_max) where each will receive the gradient

Return type:

tf.Tensor

TFUtil.safe_log(x, eps=1e-20, use_fake_grad=True)[source]

Safe wrapper around tf.log() which avoids infs or nans in the gradient.

Parameters:
  • x (tf.Tensor) –
  • eps (float|tf.Tensor) –
  • use_fake_grad (bool) – True -> use maximum_with_identity_grad, False -> use tf.maximum
Returns:

log(max(x, eps))

Return type:

tf.Tensor

TFUtil.safe_exp(x, eps=1e-20)[source]
Parameters:
  • x (tf.Tensor) –
  • eps (float) –
Returns:

exp(x), but does clipping before, such that it never returns inf nor exactly 0.0. Also, we make sure that we use the gradient in all cases.

Return type:

tf.Tensor

TFUtil.l1_normalized(x, axis=-1, eps=1e-20, use_logsumexp=False, is_not_negative=False)[source]
Parameters:
  • x (tf.Tensor) – assumes != 0
  • axis (int|tf.Tensor) – in range [-rank(x),rank(x)]
  • eps (float|tf.Tensor|None) – for safety, to ensure that tf.reduce_sum(tf.abs(x)) >= eps
  • use_logsumexp (bool) – eps must not be None
  • is_not_negative (bool) –
Returns:

y such that tf.reduce_sum(tf.abs(y)) == 1. i.e. y = x / tf.reduce_sum(tf.abs(x)).

Return type:

tf.Tensor

TFUtil.lin_exp(x, use_safe_exp=True)[source]
Parameters:
  • x (tf.Tensor) –
  • use_safe_exp (bool) –
Returns:

x + 1 if x >= 0 else exp(x). this is smooth and differentiable everywhere

Return type:

tf.Tensor

TFUtil.lin_exp_normed(x, axis=-1, eps=1e-10)[source]

This can be used as an alternative to softmax. It uses lin_exp() instead of exp.

Parameters:
  • x (tf.Tensor) –
  • axis (int|tf.Tensor) – in range [-rank(x),rank(x)]
  • eps (float|tf.Tensor|None) – for safety, to ensure that tf.reduce_sum(tf.abs(x)) >= eps
Returns:

y = l1_normalized(lin_exp(x)), i.e. tf.reduce_sum(y) == 1, and y >= 0.

Return type:

tf.Tensor

TFUtil.check_base_op_type_and_replace(x, op_type, new_op_type)[source]

Suppose you have x = tf.nn.softmax(z) and you want to get y = tf.nn.log_softmax(z). This function will test to see if x is of that kind and then return y.

Parameters:
  • x (tf.Tensor) –
  • op_type (str) – e.g. “Softmax”
  • new_op_type (str) – e.g. “LogSoftmax”
Returns:

x with new_op_type instead of op_type, or None if not matched

Return type:

tf.Tensor|None

TFUtil.copy_op(op, op_type=None, inputs=None)[source]

Copies a tf.Operation.

Parameters:
  • op (tf.Operation) –
  • op_type (str|None) –
  • inputs (list[tf.Tensor]|None) –
Returns:

copy of op but optionally change op.type == op_type or op.inputs == inputs

Return type:

tf.Operation

TFUtil.copy_tensor(x)[source]

Similar as tf.identity, but we ensure here that the return value has its own memory. This can be relevant when you want to keep a copy of the original variable value. See get_variable_value_copy_before_update_ops() for usage.

Parameters:x (tf.Tensor) –
Returns:a copy of x (points to new memory)
Return type:tf.Tensor
TFUtil.smoothing_cross_entropy(logits, labels, vocab_size, label_smoothing, gaussian=False)[source]

Cross entropy with label smoothing to limit over-confidence. Code adapted from here: https://github.com/tensorflow/tensor2tensor/blob/master/tensor2tensor/layers/common_layers.py

Parameters:
  • logits (tf.Tensor) – Tensor of size shape(labels) + [vocab_size]
  • labels (tf.Tensor) – Tensor of size […]
  • vocab_size (int) – Tensor representing the size of the vocabulary.
  • label_smoothing (float) –

    confidence = 1.0 - label_smoothing. Used to determine on and off values for label smoothing. If gaussian is true, confidence is the variance to the gaussian distribution. A common default value is 0.1. See:

  • gaussian (bool) – Uses a gaussian distribution for label smoothing
Returns:

Tensor of the same shape as labels and of the same dtype as logits.

Return type:

tf.Tensor

TFUtil.bleu_score(hypothesis, truth, hyp_seq_lens, truth_seq_lens)[source]

Calculates the BLEU score. See Util.compute_bleu(). This currently wraps a Python function and thus is not efficient.

Parameters:
  • hypothesis (tf.Tensor) – (batch, max(hyp_seq_lens))
  • truth (tf.Tensor) – (batch, max(truth_seq_lens))
  • hyp_seq_lens (tf.Tensor) – (batch,)
  • truth_seq_lens (tf.Tensor) – (batch,)
Return type:

tf.Tensor

Returns:

(batch,), float32

TFUtil.prod(ls)[source]
Parameters:ls (list[T]|tuple[T]|numpy.ndarray|tf.Tensor) –
Return type:T|int|float|tf.Tensor
class TFUtil.Lock(name='Lock')[source]

A pure TensorFlow implementation of a mutex / lock. Probably obsolete now, as with TF 1.6.0, there is tf.contrib.framework.CriticalSection.

init()[source]
lock()[source]

On first call, just returns. Any further call will block, unless there is an unlock() call.

unlock()[source]

Must be called after lock().

class TFUtil.Condition(lock=None, name='Condition')[source]

A pure TensorFlow implementation of a condition.

init()[source]
wait()[source]

Must be called with the lock held, will unlock while waiting for a signal.

wait_counter()[source]
signal()[source]

Must be called with the lock held. Emits one signal.

signal_all()[source]

Must be called with the lock held. Emits as many signals as they are waiters.

class TFUtil.GlobalTensorArrayOpMaker[source]

Creates a TensorArray which does not use the per-run (“per-step”) resource manager container but uses the standard container which persists across runs. This TensorArray resource handle is then just a standard TensorArray resource handle which can be used with all TensorArray related functions/ops.

Note: This whole implementation currently does not work because tensor_array.h is not available. See https://github.com/tensorflow/tensorflow/issues/10527 and test_GlobalTensorArray().

An alternative to this might be the MapStagingArea (https://github.com/tensorflow/tensorflow/pull/9686), which should get into TF 1.2.2.

code = '\n #include "tensorflow/core/framework/op_kernel.h"\n #include "tensorflow/core/framework/register_types.h"\n #include "tensorflow/core/framework/resource_mgr.h"\n #include "tensorflow/core/framework/tensor.h"\n #include "tensorflow/core/framework/tensor_shape.h"\n #include "tensorflow/core/framework/types.h"\n #include "tensorflow/core/kernels/bounds_check.h"\n #include "tensorflow/core/kernels/tensor_array.h"\n #include "tensorflow/core/lib/core/errors.h"\n #include "tensorflow/core/lib/core/refcount.h"\n #include "tensorflow/core/lib/strings/strcat.h"\n #include "tensorflow/core/platform/dynamic_annotations.h"\n #include "tensorflow/core/platform/logging.h"\n #include "tensorflow/core/platform/thread_annotations.h"\n #include "tensorflow/core/platform/types.h"\n\n using namespace tensorflow;\n \n // Adopted from https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/ops/data_flow_ops.cc.\n REGISTER_OP("GlobalTensorArray")\n .Input("size: int32")\n .Attr("container: string = \'\'")\n .Attr("shared_name: string = \'\'")\n .Attr("dtype: type")\n .Attr("element_shape: shape = { unknown_rank: true }")\n .Attr("dynamic_size: bool = false")\n .Attr("clear_after_read: bool = true")\n .Attr("tensor_array_name: string = \'\'")\n .Output("handle: resource")\n .Output("flow: float")\n .SetIsStateful()\n .SetShapeFn([](InferenceContext* c) {\n ShapeHandle unused;\n TF_RETURN_IF_ERROR(c->WithRank(c->input(0), 0, &unused));\n c->set_output(0, c->Vector(2));\n c->set_output(1, c->Scalar());\n return Status::OK();\n })\n .Doc("GlobalTensorArray, persistent across runs");\n \n // Copied from https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/kernels/tensor_array_ops.cc,\n // and https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/framework/resource_op_kernel.h.\n // The original TensorArrayOp used the per-run ("per-step") resource manager container\n // but we use the standard container which persists across runs.\n class GlobalTensorArrayOp : public OpKernel {\n public:\n explicit GlobalTensorArrayOp(OpKernelConstruction* context)\n : OpKernel(context), device_type_(context->device_type()) {\n OP_REQUIRES_OK(context, context->GetAttr("dtype", &dtype_));\n OP_REQUIRES_OK(context, context->GetAttr("element_shape", &element_shape_));\n OP_REQUIRES_OK(context, context->GetAttr("dynamic_size", &dynamic_size_));\n OP_REQUIRES_OK(context,\n context->GetAttr("clear_after_read", &clear_after_read_));\n OP_REQUIRES_OK(context,\n context->GetAttr("tensor_array_name", &tensor_array_name_));\n if (tensor_array_name_.empty()) tensor_array_name_ = name();\n\n AllocatorAttributes alloc_attr;\n alloc_attr.set_on_host(true);\n OP_REQUIRES_OK(context, context->allocate_persistent(\n tensorflow::DT_STRING, tensorflow::TensorShape({2}),\n &handle_, alloc_attr));\n }\n \n ~GlobalTensorArrayOp() {\n if (resource_ != nullptr) {\n resource_->Unref();\n if (cinfo_.resource_is_private_to_kernel()) {\n if (!cinfo_.resource_manager()\n ->template Delete<T>(cinfo_.container(), cinfo_.name())\n .ok()) {\n // Do nothing; the resource can have been deleted by session resets.\n }\n }\n }\n }\n \n void Compute(OpKernelContext* ctx) override {\n mutex_lock l(mu_);\n if (resource_ == nullptr) {\n ResourceMgr* mgr = ctx->resource_manager();\n OP_REQUIRES(ctx, mgr != nullptr, errors::Internal("No resource manager."));\n OP_REQUIRES_OK(ctx, cinfo_.Init(mgr, def()));\n auto h = handle_.AccessTensor(ctx)->template flat<string>();\n h(0) = cinfo_.container();\n h(1) = cinfo_.name();\n OP_REQUIRES_OK(ctx, CreateTensorArray(ctx, rm, &handle_, &resource_));\n }\n\n Tensor* handle;\n OP_REQUIRES_OK(ctx, ctx->allocate_output(0, TensorShape({}), &handle));\n handle->flat<ResourceHandle>()(0) =\n resource_->resource_handle(ctx); \n if (ctx->num_outputs() == 2) {\n // Create the flow output.\n Tensor* flow;\n OP_REQUIRES_OK(ctx, ctx->allocate_output(1, TensorShape({}), &flow));\n if (device_type_ == DEVICE_CPU) {\n // Value doesn\'t matter, but this makes msan not complaint about\n // copying an uninitialized value. To do this on GPU would require\n // a kernel launch or a host->device memcpy, so we avoid that.\n flow->flat<float>()(0) = 0;\n }\n }\n }\n \n private:\n Status CreateTensorArray(OpKernelContext* ctx, ResourceMgr* rm,\n Tensor* tensor_array_output_handle,\n TensorArray** output_tensor_array) EXCLUSIVE_LOCKS_REQUIRED(mu_) {\n const Tensor* tensor_size;\n TF_RETURN_IF_ERROR(ctx->input("size", &tensor_size));\n \n if (!TensorShapeUtils::IsScalar(tensor_size->shape())) {\n return errors::InvalidArgument(\n "TensorArray size must be scalar, but had shape: ",\n tensor_size->shape().DebugString());\n }\n const int32 size = tensor_size->scalar<int32>()();\n if (size < 0) {\n return errors::InvalidArgument("Size should be >= 0.");\n }\n \n TensorArray* tensor_array = new TensorArray(\n cinfo_.name(), dtype_, *tensor_array_output_handle, size, element_shape_,\n dynamic_size_, false /* multiple_writes_aggregate */,\n false /* is_grad */, -1 /* marked_size */, clear_after_read_);\n \n // TODO: could use LookupOrCreate instead...\n TF_RETURN_IF_ERROR(\n rm->Create(cinfo_.container(), cinfo_.name(), tensor_array));\n \n *output_tensor_array = tensor_array;\n \n return Status::OK();\n }\n\n mutex mu_;\n ContainerInfo cinfo_ GUARDED_BY(mu_);\n PersistentTensor handle_ GUARDED_BY(mu_);\n TensorArray* resource_ GUARDED_BY(mu_) = nullptr;\n \n const DeviceType device_type_;\n DataType dtype_;\n PartialTensorShape element_shape_;\n bool dynamic_size_;\n bool clear_after_read_;\n string tensor_array_name_; // The name used to create the TensorArray.\n \n TF_DISALLOW_COPY_AND_ASSIGN(GlobalTensorArrayOp);\n };\n \n REGISTER_KERNEL_BUILDER(Name("GlobalTensorArray").Device(DEVICE_CPU), GlobalTensorArrayOp);\n\n '[source]
get_op()[source]
class TFUtil.TFArrayContainer(dtype, handle=None, container=None, shared_name=None, name='array_container')[source]

Array container, like std::vector, with random index access.

Currently does not work. See https://github.com/tensorflow/tensorflow/issues/10950, and test_TFArrayContainer(). Bug #10950 is fixed upstream, should be in TF 1.2.2.

An alternative to this could be GlobalTensorArrayOpMaker and MapStagingArea, which should get into TF 1.2.2.

Parameters:
  • dtype (tf.DType) –
  • container (str) –
  • shared_name (str) –
  • name (str) –
  • handle (tf.resource) – existing handle to reuse. otherwise we will create a new one
code = '\n #include <vector>\n\n // For Eigen::ThreadPoolDevice.\n #define EIGEN_USE_THREADS 1\n\n #include "tensorflow/core/framework/op.h"\n #include "tensorflow/core/framework/shape_inference.h"\n #include "tensorflow/core/framework/op_kernel.h"\n #include "tensorflow/core/framework/resource_mgr.h"\n #include "tensorflow/core/framework/resource_op_kernel.h"\n #include "tensorflow/core/framework/tensor.h"\n #include "tensorflow/core/framework/tensor_shape.h"\n #include "tensorflow/core/framework/types.h"\n #include "tensorflow/core/platform/macros.h"\n #include "tensorflow/core/platform/mutex.h"\n #include "tensorflow/core/platform/types.h"\n #include "tensorflow/core/common_runtime/device.h"\n\n using namespace tensorflow;\n\n REGISTER_OP("ArrayContainerCreate")\n .Attr("T: type")\n .Attr("container: string = \'\'")\n .Attr("shared_name: string = \'\'")\n .Output("resource: resource")\n .SetIsStateful()\n .SetShapeFn(shape_inference::ScalarShape)\n .Doc(R"doc(Array container, random index access)doc");\n\n REGISTER_OP("ArrayContainerGetSize")\n .Input("handle: resource")\n .Output("out: int32")\n .SetShapeFn(shape_inference::ScalarShape)\n ;\n\n REGISTER_OP("ArrayContainerSetSize")\n .Input("handle: resource")\n .Input("size: int32")\n ;\n\n REGISTER_OP("ArrayContainerGet")\n .Attr("T: type")\n .Input("handle: resource")\n .Input("index: int32")\n .Output("out: T")\n ;\n\n REGISTER_OP("ArrayContainerSet")\n .Attr("T: type")\n .Input("handle: resource")\n .Input("index: int32")\n .Input("value: T")\n ;\n\n // https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/framework/resource_mgr.h\n struct ArrayContainer : public ResourceBase {\n ArrayContainer(const DataType& dtype) : dtype_(dtype) {}\n\n string DebugString() override { return "ArrayContainer"; }\n int64 MemoryUsed() const override { return 0; };\n\n mutex mu_;\n const DataType dtype_;\n std::vector<PersistentTensor> data_ GUARDED_BY(mu_);\n\n int32 get_size() {\n mutex_lock l(mu_);\n return (int32) data_.size();\n }\n\n Status set_size(int32 size) {\n if(size < 0)\n return errors::InvalidArgument("size ", size, " must be >= 0");\n mutex_lock l(mu_);\n data_.resize((size_t) size);\n return Status::OK();\n }\n\n Status get(OpKernelContext* ctx, int32 idx, PersistentTensor* v) {\n mutex_lock l(mu_);\n if(idx < 0)\n return errors::InvalidArgument("idx ", idx, " must be >= 0");\n if((size_t)idx >= data_.size())\n return errors::InvalidArgument("idx ", idx, " must be < size ", data_.size());\n PersistentTensor& t = data_[(size_t)idx];\n if(!t.IsInitialized())\n return errors::InvalidArgument("tensor at idx ", idx, " must have been set before");\n *v = t;\n return Status::OK();\n }\n\n Status set(OpKernelContext* ctx, int32 idx, const Tensor& v) {\n mutex_lock l(mu_);\n if(idx < 0)\n return errors::InvalidArgument("idx ", idx, " must be >= 0");\n if((size_t)idx >= data_.size())\n return errors::InvalidArgument("idx ", idx, " must be < size ", data_.size());\n data_[idx] = PersistentTensor(v);\n return Status::OK();\n }\n\n };\n\n // https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/framework/resource_op_kernel.h\n class ArrayContainerCreateOp : public ResourceOpKernel<ArrayContainer> {\n public:\n explicit ArrayContainerCreateOp(OpKernelConstruction* context) : ResourceOpKernel(context) {\n OP_REQUIRES_OK(context, context->GetAttr("T", &dtype_));\n }\n\n private:\n virtual bool IsCancellable() const { return false; }\n virtual void Cancel() {}\n\n Status CreateResource(ArrayContainer** ret) override EXCLUSIVE_LOCKS_REQUIRED(mu_) {\n *ret = new ArrayContainer(dtype_);\n if(*ret == nullptr)\n return errors::ResourceExhausted("Failed to allocate");\n return Status::OK();\n }\n\n Status VerifyResource(ArrayContainer* ar) override {\n if(ar->dtype_ != dtype_)\n return errors::InvalidArgument("Data type mismatch: expected ", DataTypeString(dtype_),\n " but got ", DataTypeString(ar->dtype_), ".");\n return Status::OK();\n }\n \n DataType dtype_;\n };\n REGISTER_KERNEL_BUILDER(Name("ArrayContainerCreate").Device(DEVICE_CPU), ArrayContainerCreateOp);\n\n class ArrayContainerGetSizeOp : public OpKernel {\n public:\n using OpKernel::OpKernel;\n\n void Compute(OpKernelContext* context) override {\n ArrayContainer* ar;\n \n const Tensor* handle;\n OP_REQUIRES_OK(context, context->input("handle", &handle)); \n OP_REQUIRES_OK(context, GetResourceFromContext(context, "handle", &ar));\n core::ScopedUnref unref(ar);\n\n int32 size = ar->get_size();\n Tensor* tensor_size = nullptr;\n OP_REQUIRES_OK(context, context->allocate_output(0, TensorShape({}), &tensor_size));\n tensor_size->flat<int32>().setConstant(size);\n }\n };\n REGISTER_KERNEL_BUILDER(Name("ArrayContainerGetSize").Device(DEVICE_CPU), ArrayContainerGetSizeOp);\n\n class ArrayContainerSetSizeOp : public OpKernel {\n public:\n using OpKernel::OpKernel;\n\n void Compute(OpKernelContext* context) override {\n ArrayContainer* ar;\n OP_REQUIRES_OK(context, GetResourceFromContext(context, "handle", &ar));\n core::ScopedUnref unref(ar);\n\n const Tensor* tensor_size;\n OP_REQUIRES_OK(context, context->input("size", &tensor_size));\n OP_REQUIRES(context, TensorShapeUtils::IsScalar(tensor_size->shape()),\n errors::InvalidArgument(\n "TensorArray index must be scalar, but had shape: ",\n tensor_size->shape().DebugString()));\n const int32 size = tensor_size->scalar<int32>()();\n OP_REQUIRES_OK(context, ar->set_size(size));\n }\n };\n REGISTER_KERNEL_BUILDER(Name("ArrayContainerSetSize").Device(DEVICE_CPU), ArrayContainerSetSizeOp);\n\n class ArrayContainerGetOp : public OpKernel {\n public:\n explicit ArrayContainerGetOp(OpKernelConstruction* context) : OpKernel(context) {\n OP_REQUIRES_OK(context, context->GetAttr("T", &dtype_));\n }\n\n void Compute(OpKernelContext* context) override {\n ArrayContainer* ar;\n OP_REQUIRES_OK(context, GetResourceFromContext(context, "handle", &ar));\n core::ScopedUnref unref(ar);\n\n const Tensor* tensor_index;\n OP_REQUIRES_OK(context, context->input("index", &tensor_index));\n OP_REQUIRES(context, TensorShapeUtils::IsScalar(tensor_index->shape()),\n errors::InvalidArgument(\n "TensorArray index must be scalar, but had shape: ",\n tensor_index->shape().DebugString()));\n const int32 index = tensor_index->scalar<int32>()();\n\n PersistentTensor value;\n OP_REQUIRES_OK(context, ar->get(context, index, &value));\n context->set_output(0, *value.AccessTensor(context));\n }\n\n private:\n DataType dtype_;\n };\n REGISTER_KERNEL_BUILDER(Name("ArrayContainerGet").Device(DEVICE_CPU), ArrayContainerGetOp);\n\n class ArrayContainerSetOp : public OpKernel {\n public:\n explicit ArrayContainerSetOp(OpKernelConstruction* context) : OpKernel(context) {\n OP_REQUIRES_OK(context, context->GetAttr("T", &dtype_));\n }\n\n void Compute(OpKernelContext* context) override {\n ArrayContainer* ar;\n OP_REQUIRES_OK(context, GetResourceFromContext(context, "handle", &ar));\n core::ScopedUnref unref(ar);\n\n const Tensor* tensor_index;\n const Tensor* tensor_value;\n OP_REQUIRES_OK(context, context->input("index", &tensor_index));\n OP_REQUIRES_OK(context, context->input("value", &tensor_value));\n \n OP_REQUIRES(context, TensorShapeUtils::IsScalar(tensor_index->shape()),\n errors::InvalidArgument(\n "index must be scalar, but had shape: ",\n tensor_index->shape().DebugString()));\n const int32 index = tensor_index->scalar<int32>()();\n OP_REQUIRES(context, tensor_value->IsInitialized(), errors::InvalidArgument("value must be initialized"));\n\n OP_REQUIRES_OK(context, ar->set(context, index, *tensor_value));\n }\n\n private:\n DataType dtype_;\n };\n REGISTER_KERNEL_BUILDER(Name("ArrayContainerSet").Device(DEVICE_CPU), ArrayContainerSetOp);\n '[source]
get_size()[source]
Returns:size int32 scalar
Return type:tf.Tensor
set_size(size)[source]
Parameters:size (tf.Tensor) –
Returns:operation
Return type:tf.Operation
get(index)[source]
Parameters:index (tf.Tensor) – >= 0 and < size
Returns:tensor at that index
Return type:tf.Tensor
set(index, value)[source]
Parameters:
  • index (tf.Tensor) – >= 0 and < size
  • value (tf.Tensor) –
Returns:

operation

Return type:

tf.Operation

class TFUtil.ExplicitRandomShuffleQueue(capacity, min_after_dequeue=0, dtypes=None, shapes=None, names=None, seed=None, shared_name=None, name='explicit_random_shuffle_queue')[source]

This is intended to behave very much like tf.RandomShuffleQueue, except that it’s implemented by other TF native ops / data structures, and you can change min_after_dequeue at runtime. This means that if you have your own logic about when to end, you can set min_after_dequeue=0 and dequeue all the remaining entries from the queue, and then later increase min_after_dequeue again. You can also start with a small min_after_dequeue and increase the number steadily. The original tf.RandomShuffleQueue had the effect of a reset min_after_dequeue=0 after you closed the queue. However, there was no way to reopen the queue. That is the whole reason this implementation exists.

One difference of this implementation is that you must call the init() op once before usage.

One way to implement this is in pure TF. We need some TF container type which supports having entries of different shapes (where the shape can differ where-ever we specified None). We also need some TF container which we can access by index. tf.TensorArray can handle that.

Another way to implement this is by multiple stateful tf.py_func which all reference this instance.

Parameters:
  • capacity (int) –
  • min_after_dequeue (int|tf.Tensor) –
  • dtypes (list[str|tf.DType]) –
  • shapes (list[tuple[int|tf.Tensor|None]]) –
  • names (list[str]|None) –
  • seed (int) –
  • shared_name (str|None) –
  • name (str) –
init()[source]
Return type:tf.Operation
size()[source]
Return type:tf.Tensor
min_after_dequeue_read()[source]
min_after_dequeue_assign(min_after_dequeue)[source]
Parameters:min_after_dequeue (tf.Tensor) –
Return type:tf.Operation
enqueue(v)[source]
Parameters:v (list[tf.Tensor]|dict[str,tf.Tensor]|tf.Tensor) –
Return type:tf.Operation
dequeue()[source]
TFUtil.mem_usage_for_dev(dev_name)[source]
Parameters:dev_name (str) – e.g. “/device:GPU:0” or “/job:localhost/replica:0/task:0/device:GPU:0”
Returns:int scalar, which is the peak memory usage in bytes of the given device
Return type:tf.Tensor

This function will not create multiple nodes in the graph for multiple calls. Currently only works for GPU devices.

TFUtil.identity_with_debug_log(x, args, out, name='DebugLogOp')[source]
Parameters:
  • x (tf.Tensor) –
  • args (dict[str,tf.Tensor|None]) –
  • out (list[dict[str,numpy.ndarray]]) –
  • name (str) –
Returns:

x

Return type:

tf.Tensor

TFUtil.add_check_numerics_ops(fetches=None, ignore_ops=None, use_check_numerics=True, debug_print_added_checks=True, name='add_check_numerics_ops')[source]

This is similar to tf.add_check_numerics_ops() and based on similar code. It adds some more logic and options.

Parameters:
  • fetches (list[tf.Operation|tf.Tensor]|None) – in case this is given, will only look at these and dependent ops
  • ignore_ops (list[str]) – e.g. “”
  • use_check_numerics (bool) – if False, instead of tf.check_numerics(), it does the check manually (via tf.is_finite()) and in case there is inf/nan, it will also print the tensor (while tf.check_numerics does not print the tensor). Note that this can be about 50 times slower.
  • debug_print_added_checks (bool) – prints info about each added check
  • name (str) – op-name for the final tf.group
Returns:

operation which performs all the checks

Return type:

tf.Operation

TFUtil.nested_get_shapes(x)[source]
Parameters:x (tf.Tensor|dict[str,tf.Tensor]|list[tf.Tensor]|object) – anything that nest supports
Returns:same structure as x, but tf.TensorShape for each tensor
TFUtil.has_control_flow_context(x)[source]
Parameters:x (tf.Tensor|tf.Operation|int|float|None|list[tf.Tensor|tf.Operation|int|float]) –
Returns:whether x has a control flow, i.e. is e.g. inside a while loop
Return type:bool
TFUtil.same_context(x)[source]

Will use the same (flow) context as x. E.g. if x is a constant, it can be outside the loop, so we will yield a context which is not inside the loop. See also var_creation_scope().

Parameters:x (tf.Tensor|tf.Operation|int|float|None|list[tf.Tensor|tf.Operation|int|float]) –
Returns:yields context (via tf.control_dependencies)
TFUtil.get_protobuf_fields(obj)[source]
Parameters:obj – protobuf object
Return type:dict[str]
TFUtil.get_op_attrib_keys(op)[source]
Parameters:op (tf.Operation|tf.Tensor|tf.TensorArray) –
Return type:list[str]
Returns:list of attribs. op.get_attr(key) should work
TFUtil.get_op_input_names(op)[source]

Also see: https://stackoverflow.com/questions/50723310/get-tensorflow-tf-operation-inputs-by-name

Parameters:op (tf.Operation) –
Returns:list of names with same len as op.inputs
Return type:list[str]
TFUtil.get_op_inputs_by_name(op)[source]
Parameters:op (tf.Operation) –
Returns:dict input_name -> input
Return type:dict[str,tf.Tensor]
TFUtil.tensor_array_is_dynamic_size(ta)[source]
Parameters:ta (tf.TensorArray) –
Return type:bool
TFUtil.tensor_array_is_clear_after_read(ta)[source]
Parameters:ta (tf.TensorArray) –
Return type:bool
TFUtil.tensor_array_element_shape(ta)[source]
Parameters:ta (tf.TensorArray) –
Return type:tf.TensorShape
TFUtil.tensor_array_like(ta, **kwargs)[source]
Parameters:
  • ta (tf.TensorArray) –
  • kwargs – passed to tf.TensorArray constructor
Returns:

another tensor array, just like ta

Return type:

tf.TensorArray

TFUtil.tensor_array_stack(ta, start=0, stop=None, name=None)[source]

Extends tf.TensorArray.stack by start/stop options.

Parameters:
  • ta (tf.TensorArray) –
  • start (int|tf.Tensor) –
  • stop (int|tf.Tensor|None) –
  • name (str) –
Return type:

tf.Tensor

TFUtil.select_src_beams(x, src_beams)[source]
Parameters:
  • x (tf.Tensor|tf.TensorArray|T) – (batch * src-beam, …)
  • src_beams (tf.Tensor) – (batch, beam) -> src-beam-idx
Returns:

(batch * beam, …)

Return type:

tf.Tensor|T

TFUtil.filter_ended_scores(x, end_flags, batch_dim=None, dim=None, score_zero=0.0, score_rem=-1e+30)[source]

This can e.g. used before tf.nn.top_k to let only one beam through for an ended hypothesis. Then, batch would also include the beam size, which does not matter here.

Parameters:
  • x (tf.Tensor) – (batch, dim)
  • end_flags (tf.Tensor) – (batch,)
  • batch_dim (tf.Tensor|int|None) –
  • dim (tf.Tensor|int|None) –
  • score_zero (float) – x[…, 0] will have this score where end_flag is True
  • score_rem (float) – x[…, 1:] will have this score where end_flag is False
Returns:

filtered x, (batch, dim)

Return type:

tf.Tensor

TFUtil.to_int32_64(x)[source]
Parameters:x (tf.Tensor) – dtype uint8, int8, int16, int32, int64
Return type:tf.Tensor
Returns:dtype int32 or int64
TFUtil.to_float32(x)[source]
Parameters:x (tf.Tensor) –
Returns:x as float32
Return type:tf.Tensor
TFUtil.batch_gather(x, indices, keep_dims=False)[source]
Parameters:
  • x (tf.Tensor) – (batch,dim,…)
  • indices (tf.Tensor) – (batch,) -> [0..dim-1]
  • keep_dims (bool) –
Returns:

x[batches,indices[batches]], (batch,…). or (batch,1,…) with keep_dims

Return type:

tf.Tensor

TFUtil.kernels_registered_for_op(op_name)[source]

This just wraps the TF C++ function tensorflow::KernelsRegisteredForOp().

Parameters:op_name (str) – e.g. “Gather”
Returns:e.g. [“device=’CPU’; …”, “device=’GPU’; …”, …]
Return type:list[str]
TFUtil.supported_devices_for_op(op_name)[source]
Parameters:op_name (str) –
Returns:list of devices, e.g. [“CPU”, “GPU”]
Return type:list[str]
TFUtil.find_unsupported_devices_in_graph(graph, dev_name, ignore=None)[source]
Parameters:
  • graph (tf.Graph) –
  • dev_name (str) – e.g. “GPU”
  • ignore (list[str]|None) – list of op-names to ignore, e.g. [“ScalarSummary”] etc. If None, will use defaults.
Return type:

list[tf.Operation]

TFUtil.get_device_attr(dev)[source]
Parameters:dev (str) – eg. “/device:GPU:0”, or any argument tf.device
Returns:scalar string, eg. b’device: 2, name: GeForce GTX 1080 Ti, pci bus id: 0000:82:00.0, compute capability: 6.1’
Return type:tf.Tensor
TFUtil.print_graph_output(fetches)[source]
Parameters:fetches (tf.Operation|tf.Tensor|list[tf.Tensor|tf.Operation]) –
TFUtil.find_ops_with_tensor_input(tensors, fetches=None, graph=None)[source]
Parameters:
  • tensors (tf.Tensor|tf.Variable|list[tf.Tensor]) –
  • fetches (tf.Operation|tf.Tensor|list[tf.Operation|tf.Tensor]|None) –
  • graph (tf.Graph|None) –
Returns:

list of ops

Return type:

list[tf.Operation]

TFUtil.get_var_update_ops(var, fetches=None)[source]
Parameters:
  • var (tf.Variable) –
  • fetches (tf.Operation|tf.Tensor|list[tf.Operation|tf.Tensor]|None) – e.g. the Optimizer.minimize() op
Returns:

list of ops that update var; currently expected to be of length 1

Return type:

list[tf.Operation]

TFUtil.get_variable_value_copy_before_update_ops(var, update_ops)[source]
Parameters:
  • var (tf.Variable) –
  • update_ops (list[tf.Operation]) –
Returns:

var value before any of the update_ops are executed

Return type:

tf.Tensor

TFUtil.get_variable_grad_from_update_ops(var, update_ops)[source]
Parameters:
Returns:

grad of loss w.r.t. var, as it is used in the update_ops, e.g. via ApplyAdam or ApplyGradientDescent (not all kind of updates are supported currently). If the gradient is sparse, it will return a tf.IndexedSlices.

Return type:

tf.Tensor|tf.IndexedSlices

TFUtil.add_control_input(op, control_input)[source]
Parameters:
  • op (tf.Operation) –
  • control_input (tf.Operation) –
TFUtil.bpe_idx_to_bpe_string(labels, vocab)[source]

Just does a lookup on vocab.

Parameters:
  • labels (tf.Tensor) – (batch,max_len), int32, indices in vocab
  • vocab (tf.Tensor) – (bpe_units,), string
Returns:

(batch,max_len), string

Return type:

tf.Tensor

TFUtil.string_merge(strings, seq_lens, separator=' ')[source]

Also see TFEngine.Engine.search().

Parameters:
  • strings (tf.Tensor) – (batch,max_len)
  • seq_lens (tf.Tensor) – (batch,)
  • separator (str|tf.Tensor) – string
Returns:

(batch,), string

Return type:

tf.Tensor

TFUtil.string_replace(strings, old, new, count=-1)[source]

Like str.replace.

Parameters:
  • strings (tf.Tensor) – (batch,), string
  • old (tf.Tensor|str) –
  • new (tf.Tensor|str) –
  • count (tf.Tensor|int) –
Returns:

(batch,), string

Return type:

tf.Tensor

TFUtil.bpe_merge(strings)[source]
Parameters:
  • strings (tf.Tensor) – (batch,), string
  • replace_str (tf.Tensor|str) –
Returns:

(batch,), string. strings after BPE merging

Return type:

tf.Tensor

TFUtil.words_split(strings)[source]

Basically just tf.string_split with delimiter=” “.

Parameters:strings (tf.Tensor) – (batch,), string
Returns:sparse tensor of shape (batch,max_len), string
Return type:tf.SparseTensor
TFUtil.get_sparse_tensor_length(x)[source]
Parameters:x (tf.SparseTensor) – of shape prefix + (max_len,), where prefix can be anything, e.g. prefix=(batch,)
Returns:shape prefix, int64
Return type:tf.Tensor
TFUtil.string_words_calc_wer(hyps, refs)[source]
Parameters:
  • hyps (tf.Tensor) – (batch,)
  • refs (tf.Tensor) – (batch,)
Returns:

(WER (batch,), num ref words (batch,))

Return type:

(tf.Tensor, tf.Tensor)

TFUtil.get_positional_encoding(num_channels, length=None, position=None, min_timescale=1.0, max_timescale=10000.0)[source]

Gets a bunch of sinusoids of different frequencies.

Each channel of the input Tensor is incremented by a sinusoid of a different frequency and phase.

This allows attention to learn to use absolute and relative positions. Timing signals should be added to some precursors of both the query and the memory inputs to attention.

The use of relative position is possible because sin(x+y) and cos(x+y) can be experessed in terms of y, sin(x) and cos(x).

In particular, we use a geometric sequence of timescales starting with min_timescale and ending with max_timescale. The number of different timescales is equal to channels / 2. For each timescale, we generate the two sinusoidal signals sin(timestep/timescale) and cos(timestep/timescale). All of these sinusoids are concatenated in the channels dimension.

The code is adapted from Tensor2Tensor get_timing_signal_1d (https://github.com/tensorflow/tensor2tensor).

Parameters:
  • num_channels (int) – scalar, size of timing embeddings to create. The number of different timescales is equal to channels / 2.
  • length (tf.Tensor|None) – scalar, length of timing signal sequence. if not given, is shape(position)[0]
  • position (tf.Tensor|None) – could be provided directly. int32, shape (length,)
  • min_timescale (float) – a float
  • max_timescale (float) – a float
Returns:

a Tensor of timing signals (length, channels)

Return type:

tf.Tensor