Website Blog TensorPlay documentation

Automatic differentiation package - tensorplay.autograd Automatic Mixed Precision package - tensorplay.amp Distributed communication package - tensorplay.distributed Multiprocessing package - tensorplay.multiprocessing Quantization tensorplay tensorplay.audio tensorplay.compiler tensorplay.cuda tensorplay.fft tensorplay.futures tensorplay.hub tensorplay.library tensorplay.linalg tensorplay.nn tensorplay.nn.functional tensorplay.nn.init tensorplay.optim tensorplay.random tensorplay.sparse tensorplay.special tensorplay.stax tensorplay.utils.checkpoint tensorplay.utils.data tensorplay.vision Upstream anchor labels

Reference guides

Copy

View Markdown Download .md

Distributed communication package - tensorplay.distributed

Distributed communication package - torch.distributed

`tensorplay.distributed.rendezvous.register_rendezvous_handler`	Register a new rendezvous handler.
`tensorplay.distributed.algorithms.model_averaging.utils.average_parameters`	Averages all the given parameters.
`tensorplay.distributed.algorithms.model_averaging.utils.average_parameters_or_parameter_groups`	Averages parameters of a model or parameter groups of an optimizer.
`tensorplay.distributed.algorithms.model_averaging.utils.get_params_to_average`	Return a list of parameters that need to average.

Initialization

`tensorplay.distributed.distributed_c10d.is_available`
`tensorplay.distributed.distributed_c10d.init_process_group`
`tensorplay.distributed.device_mesh.init_device_mesh`	Initializes a DeviceMesh based on device_type and mesh_shape (torch parity).
`tensorplay.distributed.distributed_c10d.is_initialized`
`tensorplay.distributed.distributed_c10d.is_mpi_available`
`tensorplay.distributed.distributed_c10d.is_nccl_available`
`tensorplay.distributed.distributed_c10d.is_gloo_available`
`tensorplay.distributed.distributed_c10d.batch_isend_irecv`	Send or receive a batch of tensors asynchronously (torch parity).
`tensorplay.distributed.distributed_c10d.destroy_process_group`
`tensorplay.distributed.distributed_c10d.irecv`	Receives a tensor asynchronously (torch parity; `src` global rank).

Post-Initialization

`tensorplay.distributed.distributed_c10d.Backend`
`tensorplay.distributed.distributed_c10d.get_backend`
`tensorplay.distributed.distributed_c10d.get_rank`
`tensorplay.distributed.distributed_c10d.get_world_size`

Groups

`tensorplay.distributed.distributed_c10d.new_group`
`tensorplay.distributed.distributed_c10d.get_group_rank`	Translate a global rank into a group rank (torch parity).
`tensorplay.distributed.distributed_c10d.get_global_rank`	Translate a group rank into a global rank (torch parity).
`tensorplay.distributed.distributed_c10d.get_process_group_ranks`	Get all ranks associated with `group` (torch parity).

DeviceMesh

tensorplay.distributed.device_mesh.DeviceMesh

DeviceMesh represents a mesh of devices (torch parity).

Point-to-point communication

`tensorplay.distributed.distributed_c10d.send`	Send a tensor synchronously (torch parity).
`tensorplay.distributed.distributed_c10d.recv`	Receives a tensor synchronously; returns the sender rank.
`tensorplay.distributed.distributed_c10d.isend`	Send a tensor asynchronously (torch parity; `dst` is a global rank).
`tensorplay.distributed.distributed_c10d.send_object_list`	Sends picklable objects in `object_list` synchronously (torch parity).
`tensorplay.distributed.distributed_c10d.recv_object_list`	Receives picklable objects in `object_list` synchronously (torch parity).
`tensorplay.distributed.distributed_c10d.P2POp`	A class to build point-to-point operations for `batch_isend_irecv`.

Collective functions

`tensorplay.distributed.distributed_c10d.broadcast`
`tensorplay.distributed.distributed_c10d.broadcast_object_list`	Broadcasts picklable objects in `object_list` to the whole group.
`tensorplay.distributed.distributed_c10d.all_reduce`
`tensorplay.distributed.distributed_c10d.reduce`
`tensorplay.distributed.distributed_c10d.all_gather`
`tensorplay.distributed.distributed_c10d.all_gather_object`	Gathers picklable objects from the whole group into a list (torch parity).
`tensorplay.distributed.distributed_c10d.gather`
`tensorplay.distributed.distributed_c10d.gather_object`	Gathers picklable objects from the whole group in a single process.
`tensorplay.distributed.distributed_c10d.scatter`
`tensorplay.distributed.distributed_c10d.scatter_object_list`	Scatters picklable objects to the whole group (torch parity).
`tensorplay.distributed.distributed_c10d.reduce_scatter`
`tensorplay.distributed.distributed_c10d.all_to_all_single`	Splits `input` evenly (or by split sizes) and scatters the chunks.
`tensorplay.distributed.distributed_c10d.all_to_all`	Scatters a list of tensors to ranks and collects one from each.
`tensorplay.distributed.distributed_c10d.barrier`
`tensorplay.distributed.distributed_c10d.Work`	Handle for an async collective (torch.distributed.Work subset).
`tensorplay.distributed.distributed_c10d.ReduceOp`

Distributed Key-Value Store

`tensorplay.distributed._store.Store`
`tensorplay.distributed._store.TCPStore`	Client for a threaded TCP key-value server (`torch.distributed.TCPStore` subset: set/get/add/delete/check as used by rendezvous and barriers).
`tensorplay.distributed._store.FileStore`	Flock-based append-log store in a single file.

Launch utility

`tensorplay.distributed.launch.launch`
`tensorplay.distributed.launch.main`
`tensorplay.distributed.launch.parse_args`

Watchdog (Experimental)

tensorplay.distributed.collective_utils.all_gather_object_enforce_type

Similar to plain all_gather_object but with additional type checking AFTER gather is done to ensure basic consistency.

Help improve this page

Found an error, an unclear step, or a missing example?

Was this page helpful?

Suggest an edit Report an issue

Automatic Mixed Precision package - tensorplay.amp

tensorplay.amp.autocast_mode.is_autocast_available

Multiprocessing package - tensorplay.multiprocessing

tensorplay.multiprocessing.reduce_tensor