returnn.util.better_exchook

https://github.com/albertz/py_better_exchook

This is a simple replacement for the standard Python exception handler (sys.excepthook). In addition to what the standard handler does, it also prints all referenced variables (no matter if local, global or builtin) of the code line of each stack frame. See below for some examples and some example output.

See these functions:

• better_exchook

• format_tb / print_tb

• iter_traceback

• get_current_frame

• dump_all_thread_tracebacks

• install

• setup_all

• replace_traceback_format_tb

• replace_traceback_print_tb

Although there might be a few more useful functions, thus we export all of them.

Also see the demo/tests at the end.

returnn.util.better_exchook.parse_py_statement(line)

Parse Python statement into tokens. Note that this is incomplete. It should be simple and fast and just barely enough for what we need here.

Reference: https://docs.python.org/3/reference/lexical_analysis.html

Parameters:

line (str)

Returns:

yields (type, value)

Return type:

Iterator[Tuple[str,str]]

returnn.util.better_exchook.parse_py_statements(source_code)

Parameters:

source_code (str)

Returns:

via parse_py_statement()

Return type:

Iterator[Tuple[str,str]]

returnn.util.better_exchook.grep_full_py_identifiers(tokens)

Parameters:

tokens (Iterable[(str,str)])

Return type:

Iterator[str]

returnn.util.better_exchook.set_linecache(filename, source)

The linecache module has some cache of the source code for the current source. Sometimes it fails to find the source of some files. We can explicitly set the source for some filename.

Parameters:

• filename (str)

• source (str)

Returns:

nothing

returnn.util.better_exchook.simple_debug_shell(globals, locals)

Parameters:

• globals (dict[str,Any])

• locals (dict[str,Any])

Returns:

nothing

returnn.util.better_exchook.debug_shell(user_ns, user_global_ns, traceback=None, execWrapper=None)

Spawns some interactive shell. Tries to use IPython if available. Falls back to pdb.post_mortem() or simple_debug_shell().

Parameters:

• user_ns (dict[str,Any])

• user_global_ns (dict[str,Any])

• traceback

• execWrapper

Returns:

nothing

returnn.util.better_exchook.output_limit()

Returns:

num chars

Return type:

int

returnn.util.better_exchook.fallback_findfile(filename)

Parameters:

filename (str)

Returns:

try to find the full filename, e.g. in modules, etc

Return type:

str|None

returnn.util.better_exchook.is_source_code_missing_brackets(source_code, prioritize_missing_open=False)

We check whether this source code snippet (e.g. one line) is complete/even w.r.t. opening/closing brackets.

Parameters:

• source_code (str)

• prioritize_missing_open (bool) – once we found any missing open bracket, directly return -1

Returns:

1 if missing_close, -1 if missing_open, 0 otherwise. I.e. whether there are missing open/close brackets. E.g. this would mean that you might want to include the prev/next source code line as well in the stack trace.

Return type:

int

returnn.util.better_exchook.is_source_code_missing_open_brackets(source_code)

We check whether this source code snippet (e.g. one line) is complete/even w.r.t. opening/closing brackets.

Parameters:

source_code (str)

Returns:

whether there are missing open brackets. E.g. this would mean that you might want to include the previous source code line as well in the stack trace.

Return type:

bool

returnn.util.better_exchook.get_source_code(filename, lineno, module_globals=None)

Parameters:

• filename (str)

• lineno (int)

• module_globals (dict[str,Any]|None)

Returns:

source code of that line (including newline)

Return type:

str

returnn.util.better_exchook.str_visible_len(s)

Parameters:

s (str)

Returns:

len without escape chars

Return type:

int

returnn.util.better_exchook.add_indent_lines(prefix, s)

Parameters:

• prefix (str)

• s (str)

Returns:

s with prefix indent added to all lines

Return type:

str

returnn.util.better_exchook.get_indent_prefix(s)

Parameters:

s (str)

Returns:

the indent spaces of s

Return type:

str

returnn.util.better_exchook.get_same_indent_prefix(lines)

Parameters:

lines (list[])

Return type:

str|None

returnn.util.better_exchook.remove_indent_lines(s)

Parameters:

s (str)

Returns:

remove as much indentation as possible

Return type:

str

returnn.util.better_exchook.replace_tab_indent(s, replace='    ')

Parameters:

• s (str) – string with tabs

• replace (str) – e.g. 4 spaces

Return type:

str

returnn.util.better_exchook.replace_tab_indents(s, replace='    ')

Parameters:

• s (str) – multi-line string with tabs

• replace (str) – e.g. 4 spaces

Return type:

str

returnn.util.better_exchook.to_bool(s, fallback=None)

Parameters:

• s (str) – str to be converted to bool, e.g. “1”, “0”, “true”, “false”

• fallback (T) – if s is not recognized as a bool

Returns:

boolean value, or fallback

Return type:

bool|T

class returnn.util.better_exchook.Color(enable=None)

Helper functions provided to perform terminal coloring.

Parameters:

enable (bool|None)

ColorIdxTable = {'black': 0, 'blue': 4, 'cyan': 6, 'green': 2, 'magenta': 5, 'red': 1, 'white': 7, 'yellow': 3}

classmethod get_global_color_enabled()

Return type:

bool

classmethod is_dark_terminal_background()

Returns:

Whether we have a dark Terminal background color, or None if unknown. We currently just check the env var COLORFGBG, which some terminals define like “<foreground-color>:<background-color>”, and if <background-color> in {0,1,2,3,4,5,6,8}, then we have some dark background. There are many other complex heuristics we could do here, which work in some cases but not in others. See e.g. here. But instead of adding more heuristics, we think that explicitly setting COLORFGBG would be the best thing, in case it’s not like you want it.

Return type:

bool|None

color(s, color=None, bold=False)

Parameters:

• s (str)

• color (str|None) – sth in self.ColorIdxTable

• bold (bool)

Returns:

s optionally wrapped with ansi escape codes

Return type:

str

py_syntax_highlight(s)

Parameters:

s (str)

Return type:

str

class returnn.util.better_exchook.DomTerm

DomTerm (https://github.com/PerBothner/DomTerm/) is a terminal emulator with many extended escape codes, such as folding text away, or even generic HTML. We can make use of some of these features (currently just folding text).

classmethod is_domterm()

Returns:

whether we are inside DomTerm

Return type:

bool

logical_block(file=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>)

Parameters:

file (io.TextIOBase|io.StringIO)

hide_button_span(mode, file=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>)

Parameters:

• mode (int) – 1 or 2

• file (io.TextIOBase|io.StringIO)

indentation(file=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>)

Parameters:

file (io.TextIOBase|io.StringIO)

hide_button(file=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>)

Parameters:

file (io.TextIOBase|io.StringIO)

fold_text_stream(prefix, postfix='', hidden_stream=None, **kwargs)

Parameters:

• prefix (str) – always visible

• postfix (str) – always visible, right after.

• hidden_stream (io.TextIOBase|io.StringIO) – sys.stdout by default. If this is sys.stdout, it will replace that stream, and collect the data during the context (in the with block).

fold_text(prefix, hidden, postfix='', file=None, align=0)

Parameters:

• prefix (str) – always visible

• hidden (str) – hidden If this is sys.stdout, it will replace that stream, and collect the data during the context (in the with block).

• postfix (str) – always visible, right after. “” by default.

• file (io.TextIOBase|io.StringIO) – sys.stdout by default.

• align (int) – remove this number of initial chars from hidden

fold_text_string(prefix, hidden, **kwargs)

Parameters:

• prefix (str)

• hidden (str)

• kwargs – passed to fold_text()

Return type:

str

returnn.util.better_exchook.is_at_exit()

Some heuristics to figure out whether this is called at a stage where the Python interpreter is shutting down.

Returns:

whether the Python interpreter is currently in the process of shutting down

Return type:

bool

returnn.util.better_exchook.format_tb(tb=None, limit=None, allLocals=None, allGlobals=None, withTitle=False, with_color=None, with_vars=None, clear_frames=True)

Parameters:

• tb (types.TracebackType|types.FrameType|StackSummary) – traceback. if None, will use sys._getframe

• limit (int|None) – limit the traceback to this number of frames. by default, will look at sys.tracebacklimit

• allLocals (dict[str,Any]|None) – if set, will update it with all locals from all frames

• allGlobals (dict[str,Any]|None) – if set, will update it with all globals from all frames

• withTitle (bool)

• with_color (bool|None) – output with ANSI escape codes for color

• with_vars (bool) – will print var content which are referenced in the source code line. by default enabled.

• clear_frames (bool) – whether to call frame.clear() after processing it. That will potentially fix some mem leaks regarding locals, so it can be important. Also see https://github.com/python/cpython/issues/113939. However, any further access to frame locals will not work (e.g. if you want to use a debugger afterwards).

Returns:

list of strings (line-based)

Return type:

list[str]

returnn.util.better_exchook.print_tb(tb, file=None, **kwargs)

Replacement for traceback.print_tb.

Parameters:

• tb (types.TracebackType|types.FrameType|StackSummary)

• file (io.TextIOBase|io.StringIO|TextIO|None) – stderr by default

Returns:

nothing, prints to file

returnn.util.better_exchook.print_exception(etype, value, tb, limit=None, file=None, chain=True)

Replacement for traceback.print_exception.

Parameters:

• etype – exception type

• value – exception value

• tb – traceback

• limit (int|None)

• file (io.TextIOBase|io.StringIO|TextIO|None) – stderr by default

• chain (bool) – whether to print the chain of exceptions

returnn.util.better_exchook.print_exc(limit=None, file=None, chain=True)

Replacement for traceback.print_exc. Shorthand for ‘print_exception(*sys.exc_info(), limit, file)’.

Parameters:

• limit (int|None)

• file (io.TextIOBase|io.StringIO|TextIO|None) – stderr by default

• chain (bool)

returnn.util.better_exchook.better_exchook(etype, value, tb, debugshell=False, autodebugshell=True, file=None, with_color=None, with_preamble=True, limit=None, chain=True)

Replacement for sys.excepthook.

Parameters:

• etype – exception type

• value – exception value

• tb – traceback

• debugshell (bool) – spawn a debug shell at the context of the exception

• autodebugshell (bool) – if env DEBUG is an integer != 0, it will spawn a debug shell

• file (io.TextIOBase|io.StringIO|TextIO|None) – output stream where we will print the traceback and exception information. stderr by default.

• with_color (bool|None) – whether to use ANSI escape codes for colored output

• with_preamble (bool) – print a short preamble for the exception

• limit (int|None)

• chain (bool) – whether to print the chain of exceptions

returnn.util.better_exchook.dump_all_thread_tracebacks(exclude_thread_ids=None, file=None)

Prints the traceback of all threads.

Parameters:

• exclude_thread_ids (set[int]|list[int]|None) – threads to exclude

• file (io.TextIOBase|io.StringIO|TextIO|None) – output stream

returnn.util.better_exchook.get_current_frame()

Returns:

current frame object (excluding this function call)

Return type:

types.FrameType

Uses sys._getframe if available, otherwise some trickery with sys.exc_info and a dummy exception.

returnn.util.better_exchook.get_func_str_from_code_object(co, frame=None)

Parameters:

• co (types.CodeType)

• frame (types.FrameType|DummyFrame|None) – if given, might provide a faster way to get the function name

Returns:

co.co_name as fallback, but maybe sth better like the full func name if possible

Return type:

str

returnn.util.better_exchook.get_func_from_code_object(co, frame=None)

Parameters:

• co (types.CodeType)

• frame (types.FrameType|DummyFrame|None) – if given, might provide a faster way to get the function name

Returns:

function, such that func.__code__ is co, or None

Return type:

types.FunctionType

This is CPython specific (to some degree; it uses the gc module to find references). Inspired from: https://stackoverflow.com/questions/12787108/getting-the-python-function-for-a-code-object https://stackoverflow.com/questions/54656758/get-function-object-from-stack-frame-object

returnn.util.better_exchook.iter_traceback(tb=None, enforce_most_recent_call_first=False)

Iterates a traceback of various formats:

• traceback (types.TracebackType)

• frame object (types.FrameType)

• stack summary (traceback.StackSummary)

Parameters:

• tb (types.TracebackType|types.FrameType|StackSummary|None) – traceback. if None, will use sys._getframe

• enforce_most_recent_call_first (bool) – Frame or stack summery: most recent call first (top of the stack is the first entry in the result) Traceback: most recent call last If True, and we get traceback, will unroll and reverse, such that we have always the most recent call first.

Returns:

yields the frames (types.FrameType)

Return type:

list[types.FrameType|DummyFrame]

class returnn.util.better_exchook.ExtendedFrameSummary(frame, **kwargs)

Extends FrameSummary by self.tb_frame.

Construct a FrameSummary.

Parameters:

• lookup_line – If True, linecache is consulted for the source code line. Otherwise, the line will be looked up when first needed.

• locals – If supplied the frame locals, which will be captured as object representations.

• line – If provided, use this instead of looking up the line in the linecache.

filename

lineno

name

locals

class returnn.util.better_exchook.DummyFrame(filename, lineno, name, f_locals=None, f_globals=None, f_builtins=None)

This class has the same attributes as a code and a frame object and is intended to be used as a dummy replacement.

classmethod from_frame_summary(f)

Parameters:

f (FrameSummary)

Return type:

DummyFrame

clear()

returnn.util.better_exchook.install()

Replaces sys.excepthook by our better_exchook.

returnn.util.better_exchook.replace_traceback_format_tb()

Replaces these functions from the traceback module by our own:

• traceback.format_tb

• traceback.StackSummary.format

• traceback.StackSummary.extract

Note that this kind of monkey patching might not be safe under all circumstances and is not officially supported by Python.

returnn.util.better_exchook.replace_traceback_print_tb()

Replaces these functions from the traceback module by our own:

• traceback.print_tb

• traceback.print_exception

• traceback.print_exc

Note that this kind of monkey patching might not be safe under all circumstances and is not officially supported by Python.

returnn.util.better_exchook.setup_all()

Calls:

• install()

• replace_traceback_format_tb()

• replace_traceback_print_tb()