returnn.torch.util.diagnose_gpu#

Diagnostic functions for GPU information, failings, memory usage, etc.

returnn.torch.util.diagnose_gpu.print_available_devices(*, file: TextIO | None = None)[source]#

Print available devices, GPU (CUDA or other), etc.

Parameters:

file – where to print to. stdout by default

returnn.torch.util.diagnose_gpu.print_using_cuda_device_report(dev: str | device, *, file: TextIO | None = None)[source]#

Theano and TensorFlow print sth like: Using gpu device 2: GeForce GTX 980 (…) Print in a similar format so that some scripts which grep our stdout work just as before.

returnn.torch.util.diagnose_gpu.diagnose_no_gpu() List[str][source]#

Diagnose why we have no GPU. Print to stdout, but also prepare summary strings.

Returns:

summary strings

returnn.torch.util.diagnose_gpu.garbage_collect()[source]#

Perform garbage collection, including any special logic for GPU.

Also see: https://github.com/pytorch/pytorch/issues/18853 https://github.com/pytorch/pytorch/issues/27600 https://pytorch.org/docs/stable/notes/faq.html#my-out-of-memory-exception-handler-can-t-allocate-memory https://github.com/Lightning-AI/pytorch-lightning/blob/7a4b0fc4331633cdf00b88776689e8a84ef96cb4/src/lightning/pytorch/utilities/memory.py#L83