returnn.util.task_system#
Here are all subprocess, threading etc related utilities, most of them quite low level.
alias of
c_int
This class provides a way to create Numpy arrays in shared memory. It adds some logic to mark whether some shared memory segment can be reused - that is when the client marks it as unused.
- Note that there are a few similar Python modules:
https://pypi.python.org/pypi/SharedArray https://parad0x.org/git/python/shared-array/about https://bitbucket.org/cleemesser/numpy-sharedmem/src https://stackoverflow.com/questions/5033799/how-do-i-pass-large-numpy-arrays https://stackoverflow.com/questions/7894791/use-numpy-array-in-shared-memory
- returnn.util.task_system.getModuleDict(modname, path=None)[source]#
- Parameters:
modname (str) – such that “import <modname>” would work
path (list[str]) – sys.path
- Returns:
the dict of the mod
- Return type:
dict[str]
- returnn.util.task_system.getModNameForModDict(obj)[source]#
- Return type:
str | None
:returns The module name or None. It will not return ‘__main__’ in any case because that likely will not be the same in the unpickling environment. Also see: https://stackoverflow.com/questions/56171796/
- returnn.util.task_system.getNormalDict(d)[source]#
- Return type:
dict[str]
It also removes getset_descriptor. New-style classes have those.
- returnn.util.task_system.assign_obj_attribs(obj, d: Dict[str, Any])[source]#
- Parameters:
obj –
d –
- Returns:
obj
Note that obj.__dict__.update(d) does not always work, e.g. when obj is a type (then obj.__dict__ is a readonly mappingproxy).
- returnn.util.task_system.numpy_set_unused(v)[source]#
- Parameters:
v (numpy.ndarray) – array which will be marked as not-used-anymore
This will tell mechanisms like SharedNumpyArray that it can reuse the memory. On the client side, this will even unmap the memory, so any further access to it will cause a SEGFAULT.
- returnn.util.task_system.numpy_copy_and_set_unused(v)[source]#
- Parameters:
v (dict[str,numpy.ndarray|object] | numpy.ndarray | object) – object to be handled
If v is a dict, we will return a new copied dict where every value is mapped through numpy_copy_and_set_unused. If v is a numpy.ndarray and its base is a SharedNumpyArray, we will copy it and
call numpy_set_unused on the old value.
If v is a numpy.ndarray and its base is not a SharedNumpyArray, we will just return it as it is and do nothing. In all other cases, we will also just return the object as it is and do nothing.
- returnn.util.task_system.numpy_alloc(shape, dtype, fortran_for_shared=False)[source]#
If EnableAutoNumpySharedMemPickling is True, this will allocate a Numpy array in shared memory so we avoid a copy later on when this Numpy array would be transferred to another process via pickling.
- class returnn.util.task_system.Pickler(*args, **kwargs)[source]#
We extend the standard Pickler to be able to pickle some more types, such as lambdas and functions, code, func cells, buffer and more.
This takes a binary file for writing a pickle data stream.
The optional protocol argument tells the pickler to use the given protocol; supported protocols are 0, 1, 2, 3, 4 and 5. The default protocol is 4. It was introduced in Python 3.4, and is incompatible with previous versions.
Specifying a negative protocol version selects the highest protocol version supported. The higher the protocol used, the more recent the version of Python needed to read the pickle produced.
The file argument must have a write() method that accepts a single bytes argument. It can thus be a file object opened for binary writing, an io.BytesIO instance, or any other custom object that meets this interface.
If fix_imports is True and protocol is less than 3, pickle will try to map the new Python 3 names to the old module names used in Python 2, so that the pickle data stream is readable with Python 2.
If buffer_callback is None (the default), buffer views are serialized into file as part of the pickle stream.
If buffer_callback is not None, then it can be called any number of times with a buffer view. If the callback returns a false value (such as None), the given buffer is out-of-band; otherwise the buffer is serialized in-band, i.e. inside the pickle stream.
It is an error if buffer_callback is not None and protocol is None or smaller than 5.
- dispatch = {<class 'NoneType'>: <function _Pickler.save_none>, <class '_io.BufferedReader'>: <function Pickler.save_iobuffer_dummy>, <class '_io.BufferedWriter'>: <function Pickler.save_iobuffer_dummy>, <class 'bool'>: <function _Pickler.save_bool>, <class 'bytearray'>: <function _Pickler.save_bytearray>, <class 'bytes'>: <function _Pickler.save_bytes>, <class 'cell'>: <function Pickler.save_cell>, <class 'code'>: <function Pickler.save_code>, <class 'dict'>: <function Pickler.intellisave_dict>, <class 'float'>: <function _Pickler.save_float>, <class 'frozenset'>: <function _Pickler.save_frozenset>, <class 'function'>: <function Pickler.save_func>, <class 'int'>: <function _Pickler.save_long>, <class 'list'>: <function _Pickler.save_list>, <class 'method'>: <function Pickler.save_method>, <class 'module'>: <function Pickler.save_module>, <class 'numpy.ndarray'>: <function Pickler.save_ndarray>, <class 'pickle.PickleBuffer'>: <function _Pickler.save_picklebuffer>, <class 'set'>: <function _Pickler.save_set>, <class 'str'>: <function Pickler.save_string>, <class 'tuple'>: <function _Pickler.save_tuple>, <class 'type'>: <function Pickler.save_type>}[source]#