# Distributed communication package - tensorplay.distributed

Source: https://www.tensorplay.cn/docs/distributed.html

# Distributed communication package - tensorplay.distributed

# Distributed communication package - torch.distributed

[tensorplay.distributed.rendezvous.register_rendezvous_handler](generated/tensorplay.distributed.rendezvous.register_rendezvous_handler.html#tensorplay.distributed.rendezvous.register_rendezvous_handler)

Register a new rendezvous handler.

[tensorplay.distributed.algorithms.model_averaging.utils.average_parameters](generated/tensorplay.distributed.algorithms.model_averaging.utils.average_parameters.html#tensorplay.distributed.algorithms.model_averaging.utils.average_parameters)

Averages all the given parameters.

[tensorplay.distributed.algorithms.model_averaging.utils.average_parameters_or_parameter_groups](generated/tensorplay.distributed.algorithms.model_averaging.utils.average_parameters_or_parameter_groups.html#tensorplay.distributed.algorithms.model_averaging.utils.average_parameters_or_parameter_groups)

Averages parameters of a model or parameter groups of an optimizer.

[tensorplay.distributed.algorithms.model_averaging.utils.get_params_to_average](generated/tensorplay.distributed.algorithms.model_averaging.utils.get_params_to_average.html#tensorplay.distributed.algorithms.model_averaging.utils.get_params_to_average)

Return a list of parameters that need to average.

## Initialization

[tensorplay.distributed.distributed_c10d.is_available](generated/tensorplay.distributed.distributed_c10d.is_available.html#tensorplay.distributed.distributed_c10d.is_available)

[tensorplay.distributed.distributed_c10d.init_process_group](generated/tensorplay.distributed.distributed_c10d.init_process_group.html#tensorplay.distributed.distributed_c10d.init_process_group)

[tensorplay.distributed.device_mesh.init_device_mesh](generated/tensorplay.distributed.device_mesh.init_device_mesh.html#tensorplay.distributed.device_mesh.init_device_mesh)

Initializes a DeviceMesh based on device_type and mesh_shape (torch parity).

[tensorplay.distributed.distributed_c10d.is_initialized](generated/tensorplay.distributed.distributed_c10d.is_initialized.html#tensorplay.distributed.distributed_c10d.is_initialized)

[tensorplay.distributed.distributed_c10d.is_mpi_available](generated/tensorplay.distributed.distributed_c10d.is_mpi_available.html#tensorplay.distributed.distributed_c10d.is_mpi_available)

[tensorplay.distributed.distributed_c10d.is_nccl_available](generated/tensorplay.distributed.distributed_c10d.is_nccl_available.html#tensorplay.distributed.distributed_c10d.is_nccl_available)

[tensorplay.distributed.distributed_c10d.is_gloo_available](generated/tensorplay.distributed.distributed_c10d.is_gloo_available.html#tensorplay.distributed.distributed_c10d.is_gloo_available)

[tensorplay.distributed.distributed_c10d.batch_isend_irecv](generated/tensorplay.distributed.distributed_c10d.batch_isend_irecv.html#tensorplay.distributed.distributed_c10d.batch_isend_irecv)

Send or receive a batch of tensors asynchronously (torch parity).

[tensorplay.distributed.distributed_c10d.destroy_process_group](generated/tensorplay.distributed.distributed_c10d.destroy_process_group.html#tensorplay.distributed.distributed_c10d.destroy_process_group)

[tensorplay.distributed.distributed_c10d.irecv](generated/tensorplay.distributed.distributed_c10d.irecv.html#tensorplay.distributed.distributed_c10d.irecv)

Receives a tensor asynchronously (torch parity; src global rank).

## Post-Initialization

[tensorplay.distributed.distributed_c10d.Backend](generated/tensorplay.distributed.distributed_c10d.Backend.html#tensorplay.distributed.distributed_c10d.Backend)

[tensorplay.distributed.distributed_c10d.get_backend](generated/tensorplay.distributed.distributed_c10d.get_backend.html#tensorplay.distributed.distributed_c10d.get_backend)

[tensorplay.distributed.distributed_c10d.get_rank](generated/tensorplay.distributed.distributed_c10d.get_rank.html#tensorplay.distributed.distributed_c10d.get_rank)

[tensorplay.distributed.distributed_c10d.get_world_size](generated/tensorplay.distributed.distributed_c10d.get_world_size.html#tensorplay.distributed.distributed_c10d.get_world_size)

## Groups

[tensorplay.distributed.distributed_c10d.new_group](generated/tensorplay.distributed.distributed_c10d.new_group.html#tensorplay.distributed.distributed_c10d.new_group)

[tensorplay.distributed.distributed_c10d.get_group_rank](generated/tensorplay.distributed.distributed_c10d.get_group_rank.html#tensorplay.distributed.distributed_c10d.get_group_rank)

Translate a global rank into a group rank (torch parity).

[tensorplay.distributed.distributed_c10d.get_global_rank](generated/tensorplay.distributed.distributed_c10d.get_global_rank.html#tensorplay.distributed.distributed_c10d.get_global_rank)

Translate a group rank into a global rank (torch parity).

[tensorplay.distributed.distributed_c10d.get_process_group_ranks](generated/tensorplay.distributed.distributed_c10d.get_process_group_ranks.html#tensorplay.distributed.distributed_c10d.get_process_group_ranks)

Get all ranks associated with group (torch parity).

## DeviceMesh

[tensorplay.distributed.device_mesh.DeviceMesh](generated/tensorplay.distributed.device_mesh.DeviceMesh.html#tensorplay.distributed.device_mesh.DeviceMesh)

DeviceMesh represents a mesh of devices (torch parity).

## Point-to-point communication

[tensorplay.distributed.distributed_c10d.send](generated/tensorplay.distributed.distributed_c10d.send.html#tensorplay.distributed.distributed_c10d.send)

Send a tensor synchronously (torch parity).

[tensorplay.distributed.distributed_c10d.recv](generated/tensorplay.distributed.distributed_c10d.recv.html#tensorplay.distributed.distributed_c10d.recv)

Receives a tensor synchronously; returns the sender rank.

[tensorplay.distributed.distributed_c10d.isend](generated/tensorplay.distributed.distributed_c10d.isend.html#tensorplay.distributed.distributed_c10d.isend)

Send a tensor asynchronously (torch parity; dst is a global rank).

[tensorplay.distributed.distributed_c10d.send_object_list](generated/tensorplay.distributed.distributed_c10d.send_object_list.html#tensorplay.distributed.distributed_c10d.send_object_list)

Sends picklable objects in object_list synchronously (torch parity).

[tensorplay.distributed.distributed_c10d.recv_object_list](generated/tensorplay.distributed.distributed_c10d.recv_object_list.html#tensorplay.distributed.distributed_c10d.recv_object_list)

Receives picklable objects in object_list synchronously (torch parity).

[tensorplay.distributed.distributed_c10d.P2POp](generated/tensorplay.distributed.distributed_c10d.P2POp.html#tensorplay.distributed.distributed_c10d.P2POp)

A class to build point-to-point operations for batch_isend_irecv.

## Collective functions

[tensorplay.distributed.distributed_c10d.broadcast](generated/tensorplay.distributed.distributed_c10d.broadcast.html#tensorplay.distributed.distributed_c10d.broadcast)

[tensorplay.distributed.distributed_c10d.broadcast_object_list](generated/tensorplay.distributed.distributed_c10d.broadcast_object_list.html#tensorplay.distributed.distributed_c10d.broadcast_object_list)

Broadcasts picklable objects in object_list to the whole group.

[tensorplay.distributed.distributed_c10d.all_reduce](generated/tensorplay.distributed.distributed_c10d.all_reduce.html#tensorplay.distributed.distributed_c10d.all_reduce)

[tensorplay.distributed.distributed_c10d.reduce](generated/tensorplay.distributed.distributed_c10d.reduce.html#tensorplay.distributed.distributed_c10d.reduce)

[tensorplay.distributed.distributed_c10d.all_gather](generated/tensorplay.distributed.distributed_c10d.all_gather.html#tensorplay.distributed.distributed_c10d.all_gather)

[tensorplay.distributed.distributed_c10d.all_gather_object](generated/tensorplay.distributed.distributed_c10d.all_gather_object.html#tensorplay.distributed.distributed_c10d.all_gather_object)

Gathers picklable objects from the whole group into a list (torch parity).

[tensorplay.distributed.distributed_c10d.gather](generated/tensorplay.distributed.distributed_c10d.gather.html#tensorplay.distributed.distributed_c10d.gather)

[tensorplay.distributed.distributed_c10d.gather_object](generated/tensorplay.distributed.distributed_c10d.gather_object.html#tensorplay.distributed.distributed_c10d.gather_object)

Gathers picklable objects from the whole group in a single process.

[tensorplay.distributed.distributed_c10d.scatter](generated/tensorplay.distributed.distributed_c10d.scatter.html#tensorplay.distributed.distributed_c10d.scatter)

[tensorplay.distributed.distributed_c10d.scatter_object_list](generated/tensorplay.distributed.distributed_c10d.scatter_object_list.html#tensorplay.distributed.distributed_c10d.scatter_object_list)

Scatters picklable objects to the whole group (torch parity).

[tensorplay.distributed.distributed_c10d.reduce_scatter](generated/tensorplay.distributed.distributed_c10d.reduce_scatter.html#tensorplay.distributed.distributed_c10d.reduce_scatter)

[tensorplay.distributed.distributed_c10d.all_to_all_single](generated/tensorplay.distributed.distributed_c10d.all_to_all_single.html#tensorplay.distributed.distributed_c10d.all_to_all_single)

Splits input evenly (or by split sizes) and scatters the chunks.

[tensorplay.distributed.distributed_c10d.all_to_all](generated/tensorplay.distributed.distributed_c10d.all_to_all.html#tensorplay.distributed.distributed_c10d.all_to_all)

Scatters a list of tensors to ranks and collects one from each.

[tensorplay.distributed.distributed_c10d.barrier](generated/tensorplay.distributed.distributed_c10d.barrier.html#tensorplay.distributed.distributed_c10d.barrier)

[tensorplay.distributed.distributed_c10d.Work](generated/tensorplay.distributed.distributed_c10d.Work.html#tensorplay.distributed.distributed_c10d.Work)

Handle for an async collective (torch.distributed.Work subset).

[tensorplay.distributed.distributed_c10d.ReduceOp](generated/tensorplay.distributed.distributed_c10d.ReduceOp.html#tensorplay.distributed.distributed_c10d.ReduceOp)

## Distributed Key-Value Store

[tensorplay.distributed._store.Store](generated/tensorplay.distributed._store.Store.html#tensorplay.distributed._store.Store)

[tensorplay.distributed._store.TCPStore](generated/tensorplay.distributed._store.TCPStore.html#tensorplay.distributed._store.TCPStore)

Client for a threaded TCP key-value server (torch.distributed.TCPStore subset: set/get/add/delete/check as used by rendezvous and barriers).

[tensorplay.distributed._store.FileStore](generated/tensorplay.distributed._store.FileStore.html#tensorplay.distributed._store.FileStore)

Flock-based append-log store in a single file.

## Launch utility

[tensorplay.distributed.launch.launch](generated/tensorplay.distributed.launch.launch.html#tensorplay.distributed.launch.launch)

[tensorplay.distributed.launch.main](generated/tensorplay.distributed.launch.main.html#tensorplay.distributed.launch.main)

[tensorplay.distributed.launch.parse_args](generated/tensorplay.distributed.launch.parse_args.html#tensorplay.distributed.launch.parse_args)

## Watchdog (Experimental)

[tensorplay.distributed.collective_utils.all_gather_object_enforce_type](generated/tensorplay.distributed.collective_utils.all_gather_object_enforce_type.html#tensorplay.distributed.collective_utils.all_gather_object_enforce_type)

Similar to plain all_gather_object but with additional type checking AFTER gather is done to ensure basic consistency.