returnn.tf.util.ken_lm#

Uses KenLM (https://kheafield.com/code/kenlm/) (extern/kenlm) to read n-gram LMs (ARPA format), and provides a TF op to use them.

returnn.tf.util.ken_lm.kenlm_checked_out()[source]#
Return type:

bool

returnn.tf.util.ken_lm.get_tf_mod(verbose=False)[source]#
Parameters:

verbose (bool) –

Returns:

module

returnn.tf.util.ken_lm.ken_lm_load(filename)[source]#
Parameters:

filename (str) –

Returns:

TF resource handle

Return type:

tf.Tensor

returnn.tf.util.ken_lm.ken_lm_abs_score_strings(handle, strings)[source]#
Parameters:

  • handle (tf.Tensor) – TF resource handle returned by ken_lm_load()

  • strings (tf.Tensor) – strings which are being scores. white-space delimited words.

Returns:

same shape as strings, float32

Return type:

tf.Tensor

returnn.tf.util.ken_lm.ken_lm_abs_score_bpe_strings(handle, bpe_merge_symbol, strings)[source]#
Parameters:

  • handle (tf.Tensor) – TF resource handle returned by ken_lm_load()

  • bpe_merge_symbol (str) – e.g. “@@”

  • strings (tf.Tensor) – strings which are being scores. white-space delimited words.

Returns:

same shape as strings, float32

Return type:

tf.Tensor

returnn.tf.util.ken_lm.ken_lm_abs_score_bpe_strings_dense(handle, bpe_merge_symbol, strings, labels)[source]#
Parameters:

  • handle (tf.Tensor) – TF resource handle returned by ken_lm_load()

  • bpe_merge_symbol (str) – e.g. “@@”

  • strings (tf.Tensor) – strings which are being scores. white-space delimited words.

  • labels (tf.Tensor|tf.Variable) –

Returns:

same shape as strings, float32

Return type:

tf.Tensor