returnn.torch.data.extern_data
¶
From raw dict to extern_data tensor dict.
- returnn.torch.data.extern_data.extern_data_template_from_config_opts(extern_data_dict: Dict[str, Any]) TensorDict [source]¶
- Parameters:
extern_data_dict – as you would specify in the config
- Returns:
extern data tensor dict
- returnn.torch.data.extern_data.raw_dict_to_extern_data(extern_data_raw: Dict[str, Tensor | ndarray], *, extern_data_template: TensorDict, device: str | device, float_dtype: str | dtype | None = None, with_eval_targets: bool = False) TensorDict [source]¶
- Parameters:
extern_data_raw – This comes out of the DataLoader, via our collate_batch.
extern_data_template – Specified via extern_data in the config.
device – E.g. the GPU.
float_dtype
with_eval_targets – if False, we skip all tensors with
available_for_inference=False
.
- Returns:
tensor dict, like extern_data_template, but with raw tensors set to Torch tensors, on the right device.
- returnn.torch.data.extern_data.raw_dict_can_split_batch(extern_data_raw: Dict[str, Tensor | ndarray], *, num_splits: int = 2) bool [source]¶
- Parameters:
extern_data_raw – This comes out of the DataLoader, via our collate_batch.
num_splits – into how many parts
- Returns:
- returnn.torch.data.extern_data.raw_dict_split_batch(extern_data_raw: Dict[str, Tensor | ndarray], *, splits: Sequence[int] | int) List[Dict[str, Tensor | ndarray]] [source]¶
- Parameters:
extern_data_raw – This comes out of the DataLoader, via our collate_batch.
splits – either the list of resulting batch dims, or the num of splits
- Returns:
splitted extern_data_raw
- returnn.torch.data.extern_data.get_batch_dim_from_extern_data(extern_data: TensorDict) Dim [source]¶
We expect that the batch dim is the first dim in any of the tensors. See collate_batch.
We allow that the batch dim is not necessarily the global batch_dim object. We also allow that this is not even marked as batch dim (is_batch_dim() can be False).