TFKenLM

Uses KenLM (http://kheafield.com/code/kenlm/) (extern/kenlm) to read n-gram LMs (ARPA format), and provides a TF op to use them.

TFKenLM.get_tf_mod(verbose=False)[source]
TFKenLM.ken_lm_load(filename)[source]
Parameters:filename (str) –
Returns:TF resource handle
Return type:tf.Tensor
TFKenLM.ken_lm_abs_score_strings(handle, strings)[source]
Parameters:
  • handle (tf.Tensor) – TF resource handle returned by ken_lm_load()
  • strings (tf.Tensor) – strings which are being scores. white-space delimited words.
Returns:

same shape as strings, float32

Return type:

tf.Tensor

TFKenLM.ken_lm_abs_score_bpe_strings(handle, bpe_merge_symbol, strings)[source]
Parameters:
  • handle (tf.Tensor) – TF resource handle returned by ken_lm_load()
  • bpe_merge_symbol (str) – e.g. “@@”
  • strings (tf.Tensor) – strings which are being scores. white-space delimited words.
Returns:

same shape as strings, float32

Return type:

tf.Tensor

TFKenLM.ken_lm_abs_score_bpe_strings_dense(handle, bpe_merge_symbol, strings, labels)[source]
Parameters:
  • handle (tf.Tensor) – TF resource handle returned by ken_lm_load()
  • bpe_merge_symbol (str) – e.g. “@@”
  • strings (tf.Tensor) – strings which are being scores. white-space delimited words.
  • labels (tf.Tensor|tf.Variable) –
Returns:

same shape as strings, float32

Return type:

tf.Tensor