# TensorPlay documentation Source: https://www.tensorplay.cn/docs/ # TensorPlay documentation TensorPlay is a PyTorch-compatible tensor library for deep learning using GPUs and CPUs. Features described in this documentation are classified by release status: Stable (API-Stable): These features will be maintained long-term and there should generally be no major performance limitations or gaps in documentation. We also expect to maintain backwards compatibility (although breaking changes can happen and notice will be given one release ahead of time). Unstable (API-Unstable): Encompasses all features that are under active development where APIs may change based on user feedback, requisite performance improvements or because coverage across operators is not yet complete. The APIs and performance characteristics of these features may change. - [tensorplay](tensorplay.html) [Tensors](tensorplay.html#tensors) - [Generators](tensorplay.html#generators) - [Random sampling](tensorplay.html#random-sampling) - [Serialization](tensorplay.html#serialization) - [Parallelism](tensorplay.html#parallelism) - [Locally disabling gradient computation](tensorplay.html#locally-disabling-gradient-computation) - [Utilities](tensorplay.html#utilities) - [Optimizations](tensorplay.html#optimizations) - [TensorPlay-specific additions](tensorplay.html#tensorplay-specific-additions) - [Automatic differentiation package - tensorplay.autograd](autograd.html) - [Automatic differentiation package - torch.autograd](autograd.html#automatic-differentiation-package-torch-autograd) [tensorplay._C._autograd.backward](generated/tensorplay._C._autograd.backward.html) - [tensorplay.autograd.grad](generated/tensorplay.autograd.grad.html) - [Functional higher level API](autograd.html#functional-higher-level-api) - [Function](autograd.html#function) - [Custom Function utilities](autograd.html#custom-function-utilities) - [Numerical gradient checking](autograd.html#numerical-gradient-checking) - [Debugging and anomaly detection](autograd.html#debugging-and-anomaly-detection) - [tensorplay.nn](nn.html) [Containers](nn.html#containers) - [Convolution Layers](nn.html#convolution-layers) - [Pooling layers](nn.html#pooling-layers) - [Padding Layers](nn.html#padding-layers) - [Non-linear Activations (weighted sum, nonlinearity)](nn.html#non-linear-activations-weighted-sum-nonlinearity) - [Non-linear Activations (other)](nn.html#non-linear-activations-other) - [Normalization Layers](nn.html#normalization-layers) - [Recurrent Layers](nn.html#recurrent-layers) - [Transformer Layers](nn.html#transformer-layers) - [Linear Layers](nn.html#linear-layers) - [Dropout Layers](nn.html#dropout-layers) - [Sparse Layers](nn.html#sparse-layers) - [Distance Functions](nn.html#distance-functions) - [Loss Functions](nn.html#loss-functions) - [Vision Layers](nn.html#vision-layers) - [Shuffle Layers](nn.html#shuffle-layers) - [DataParallel Layers (multi-GPU, distributed)](nn.html#dataparallel-layers-multi-gpu-distributed) - [Utilities](nn.html#utilities) - [Lazy Modules Initialization](nn.html#lazy-modules-initialization) - [TensorPlay-specific additions](nn.html#tensorplay-specific-additions) - [tensorplay.nn.functional](nn.functional.html) [Convolution functions](nn.functional.html#convolution-functions) - [Pooling functions](nn.functional.html#pooling-functions) - [Attention Mechanisms](nn.functional.html#attention-mechanisms) - [Non-linear activation functions](nn.functional.html#non-linear-activation-functions) - [Linear functions](nn.functional.html#linear-functions) - [Dropout functions](nn.functional.html#dropout-functions) - [Sparse functions](nn.functional.html#sparse-functions) - [Distance functions](nn.functional.html#distance-functions) - [Loss functions](nn.functional.html#loss-functions) - [Vision functions](nn.functional.html#vision-functions) - [Low-Precision functions](nn.functional.html#low-precision-functions) - [TensorPlay-specific additions](nn.functional.html#tensorplay-specific-additions) - [tensorplay.nn.init](nn.init.html) - [torch.nn.init](nn.init.html#torch-nn-init) [tensorplay.nn.init.calculate_gain](generated/tensorplay.nn.init.calculate_gain.html) - [tensorplay.nn.init.uniform_](generated/tensorplay.nn.init.uniform_.html) - [tensorplay.nn.init.normal_](generated/tensorplay.nn.init.normal_.html) - [tensorplay.nn.init.constant_](generated/tensorplay.nn.init.constant_.html) - [tensorplay.nn.init.ones_](generated/tensorplay.nn.init.ones_.html) - [tensorplay.nn.init.zeros_](generated/tensorplay.nn.init.zeros_.html) - [tensorplay.nn.init.eye_](generated/tensorplay.nn.init.eye_.html) - [tensorplay.nn.init.dirac_](generated/tensorplay.nn.init.dirac_.html) - [tensorplay.nn.init.xavier_uniform_](generated/tensorplay.nn.init.xavier_uniform_.html) - [tensorplay.nn.init.xavier_normal_](generated/tensorplay.nn.init.xavier_normal_.html) - [tensorplay.nn.init.kaiming_uniform_](generated/tensorplay.nn.init.kaiming_uniform_.html) - [tensorplay.nn.init.kaiming_normal_](generated/tensorplay.nn.init.kaiming_normal_.html) - [tensorplay.nn.init.trunc_normal_](generated/tensorplay.nn.init.trunc_normal_.html) - [tensorplay.nn.init.orthogonal_](generated/tensorplay.nn.init.orthogonal_.html) - [tensorplay.nn.init.sparse_](generated/tensorplay.nn.init.sparse_.html) - [tensorplay.optim](optim.html) [Base class](optim.html#base-class) - [Module-level hooks](optim.html#module-level-hooks) - [Utilities](optim.html#utilities) - [Algorithms](optim.html#algorithms) - [How to adjust learning rate](optim.html#how-to-adjust-learning-rate) - [tensorplay.cuda](cuda.html) - [torch.cuda](cuda.html#torch-cuda) [StreamContext](generated/tensorplay.cuda.StreamContext.html) - [tensorplay.cuda.can_device_access_peer](generated/tensorplay.cuda.can_device_access_peer.html) - [tensorplay.cuda.check_error](generated/tensorplay.cuda.check_error.html) - [tensorplay.cuda.current_blas_handle](generated/tensorplay.cuda.current_blas_handle.html) - [tensorplay.cuda.current_solver_handle](generated/tensorplay.cuda.current_solver_handle.html) - [tensorplay.cuda.current_device](generated/tensorplay.cuda.current_device.html) - [tensorplay.cuda.current_stream](generated/tensorplay.cuda.current_stream.html) - [tensorplay.cuda.cudart](generated/tensorplay.cuda.cudart.html) - [tensorplay.cuda.default_stream](generated/tensorplay.cuda.default_stream.html) - [device](generated/tensorplay.cuda.device.html) - [tensorplay.cuda.device_count](generated/tensorplay.cuda.device_count.html) - [tensorplay.cuda.device_memory_used](generated/tensorplay.cuda.device_memory_used.html) - [device_of](generated/tensorplay.cuda.device_of.html) - [tensorplay.cuda.get_arch_list](generated/tensorplay.cuda.get_arch_list.html) - [tensorplay.cuda.get_device_capability](generated/tensorplay.cuda.get_device_capability.html) - [tensorplay.cuda.get_device_name](generated/tensorplay.cuda.get_device_name.html) - [tensorplay.cuda.get_device_properties](generated/tensorplay.cuda.get_device_properties.html) - [tensorplay.cuda.get_gencode_flags](generated/tensorplay.cuda.get_gencode_flags.html) - [tensorplay.cuda.get_stream_from_external](generated/tensorplay.cuda.get_stream_from_external.html) - [tensorplay.cuda.get_sync_debug_mode](generated/tensorplay.cuda.get_sync_debug_mode.html) - [tensorplay.cuda.init](generated/tensorplay.cuda.init.html) - [tensorplay.cuda.ipc_collect](generated/tensorplay.cuda.ipc_collect.html) - [tensorplay.cuda.is_available](generated/tensorplay.cuda.is_available.html) - [tensorplay.cuda.is_bf16_supported](generated/tensorplay.cuda.is_bf16_supported.html) - [tensorplay.cuda.is_initialized](generated/tensorplay.cuda.is_initialized.html) - [tensorplay.cuda.is_tf32_supported](generated/tensorplay.cuda.is_tf32_supported.html) - [tensorplay.cuda.memory_usage](generated/tensorplay.cuda.memory_usage.html) - [tensorplay.cuda.set_device](generated/tensorplay.cuda.set_device.html) - [tensorplay.cuda.set_stream](generated/tensorplay.cuda.set_stream.html) - [tensorplay.cuda.set_sync_debug_mode](generated/tensorplay.cuda.set_sync_debug_mode.html) - [tensorplay.cuda.stream](generated/tensorplay.cuda.stream.html) - [tensorplay.cuda.synchronize](generated/tensorplay.cuda.synchronize.html) - [tensorplay.cuda.utilization](generated/tensorplay.cuda.utilization.html) - [tensorplay.cuda.temperature](generated/tensorplay.cuda.temperature.html) - [tensorplay.cuda.power_draw](generated/tensorplay.cuda.power_draw.html) - [tensorplay.cuda.clock_rate](generated/tensorplay.cuda.clock_rate.html) - [tensorplay.cuda.AcceleratorError](generated/tensorplay.cuda.AcceleratorError.html) - [tensorplay.cuda.OutOfMemoryError](generated/tensorplay.cuda.OutOfMemoryError.html) - [Random Number Generator](cuda.html#random-number-generator) - [Streams and events](cuda.html#streams-and-events) - [Graphs (beta)](cuda.html#graphs-beta) - [Graph Kernel Annotations (prototype)](cuda.html#graph-kernel-annotations-prototype) - [Memory management](cuda.html#memory-management) - [NVIDIA Tools Extension (NVTX)](cuda.html#nvidia-tools-extension-nvtx) - [GPUDirect Storage (prototype)](cuda.html#gpudirect-storage-prototype) - [Green Contexts (experimental)](cuda.html#green-contexts-experimental) - [TensorPlay-specific additions](cuda.html#tensorplay-specific-additions) - [Automatic Mixed Precision package - tensorplay.amp](amp.html) [Autocasting](amp.html#autocasting) - [Gradient Scaling](amp.html#gradient-scaling) - [tensorplay.linalg](linalg.html) [Matrix Properties](linalg.html#matrix-properties) - [Decompositions](linalg.html#decompositions) - [Solvers](linalg.html#solvers) - [Inverses](linalg.html#inverses) - [Matrix Functions](linalg.html#matrix-functions) - [Matrix Products](linalg.html#matrix-products) - [Tensor Operations](linalg.html#tensor-operations) - [Misc](linalg.html#misc) - [Experimental Functions](linalg.html#experimental-functions) - [tensorplay.fft](fft.html) [Fast Fourier Transforms](fft.html#fast-fourier-transforms) - [Helper Functions](fft.html#helper-functions) - [tensorplay.special](special.html) [Functions](special.html#functions) - [tensorplay.sparse](sparse.html) [Tensor methods and sparse](sparse.html#tensor-methods-and-sparse) - [tensorplay.random](random.html) [Random Generator](random.html#random-generator) - [tensorplay.utils.data](data.html) [Memory Pinning](data.html#memory-pinning) - [tensorplay.utils.checkpoint](checkpoint.html) - [torch.utils.checkpoint](checkpoint.html#torch-utils-checkpoint) [tensorplay.utils.checkpoint.checkpoint](generated/tensorplay.utils.checkpoint.checkpoint.html) - [tensorplay.utils.checkpoint.checkpoint_sequential](generated/tensorplay.utils.checkpoint.checkpoint_sequential.html) - [CheckpointPolicy](generated/tensorplay.utils.checkpoint.CheckpointPolicy.html) - [tensorplay.utils.checkpoint.set_checkpoint_early_stop](generated/tensorplay.utils.checkpoint.set_checkpoint_early_stop.html) - [tensorplay.futures](futures.html) - [torch.futures](futures.html#torch-futures) [Future](generated/tensorplay.futures.Future.html) - [tensorplay.hub](hub.html) [Loading models from Hub](hub.html#loading-models-from-hub) - [Multiprocessing package - tensorplay.multiprocessing](multiprocessing.html) [API Reference](multiprocessing.html#api-reference) - [tensorplay.library](library.html) [Choosing the kind of custom op](library.html#choosing-the-kind-of-custom-op) - [Extending custom ops (created from Python or C++)](library.html#extending-custom-ops-created-from-python-or-c) - [Low-level APIs](library.html#low-level-apis) - [Distributed communication package - tensorplay.distributed](distributed.html) - [Distributed communication package - torch.distributed](distributed.html#distributed-communication-package-torch-distributed) [tensorplay.distributed.rendezvous.register_rendezvous_handler](generated/tensorplay.distributed.rendezvous.register_rendezvous_handler.html) - [tensorplay.distributed.algorithms.model_averaging.utils.average_parameters](generated/tensorplay.distributed.algorithms.model_averaging.utils.average_parameters.html) - [tensorplay.distributed.algorithms.model_averaging.utils.average_parameters_or_parameter_groups](generated/tensorplay.distributed.algorithms.model_averaging.utils.average_parameters_or_parameter_groups.html) - [tensorplay.distributed.algorithms.model_averaging.utils.get_params_to_average](generated/tensorplay.distributed.algorithms.model_averaging.utils.get_params_to_average.html) - [Initialization](distributed.html#initialization) - [Post-Initialization](distributed.html#post-initialization) - [Groups](distributed.html#groups) - [DeviceMesh](distributed.html#devicemesh) - [Point-to-point communication](distributed.html#point-to-point-communication) - [Collective functions](distributed.html#collective-functions) - [Distributed Key-Value Store](distributed.html#distributed-key-value-store) - [Launch utility](distributed.html#launch-utility) - [Watchdog (Experimental)](distributed.html#watchdog-experimental) - [Quantization](quantization.html) [Quantization API Reference (Kept since APIs are still public)](quantization.html#quantization-api-reference-kept-since-apis-are-still-public) - [tensorplay.compiler](compiler.html) [Graph IR](compiler.html#graph-ir) - [CUDA graphs](compiler.html#cuda-graphs) - [Caching and errors](compiler.html#caching-and-errors) - [Related modules](compiler.html#related-modules) - [tensorplay.stax](stax.html) [Functions](stax.html#functions) - [tensorplay.vision](vision.html) [Top-level functions](vision.html#top-level-functions) - [Submodules](vision.html#submodules) - [tensorplay.audio](audio.html) [Top-level functions](audio.html#top-level-functions) - [Submodules](audio.html#submodules) ## Indices and tables - [Index](genindex.html) - [Module Index](py-modindex.html) - [Search Page](search.html)