# tensorplay.distributed.algorithms API

Source: https://www.tensorplay.cn/docs/api/tensorplay.distributed.algorithms.html

## Classes 3

[#](#api-tensorplay.distributed.algorithms.Join)

### Join

class[Full reference ↗](/docs/generated/tensorplay.distributed.algorithms.Join.html)

```python
class tensorplay.distributed.algorithms.Join(joinables: list[Joinable], enable: bool = True, throw_on_early_termination: bool = False, **kwargs)
```

This class defines the generic join context manager, which allows custom hooks to be called after a process joins.

These hooks should shadow the
collective communications of non-joined processes to prevent hanging and
erroring and to ensure algorithmic correctness. Refer to [JoinHook](/docs/generated/tensorplay.distributed.algorithms.JoinHook.html#tensorplay.distributed.algorithms.JoinHook)
for details about the hook definition.

> **Warning**
>
> The context manager requires each participating Joinable to call the method notify_join_context() before its own per- iteration collective communications to ensure correctness.

Parameters:

- joinables (List[[Joinable](/docs/generated/tensorplay.distributed.algorithms.Joinable.html#tensorplay.distributed.algorithms.Joinable)]) – a list of the participating [Joinable](/docs/generated/tensorplay.distributed.algorithms.Joinable.html#tensorplay.distributed.algorithms.Joinable) s; their hooks are iterated over in the given order.

- enable ([bool](https://docs.python.org/3/builtins/functions.html#bool)) – a flag enabling uneven input detection; setting to False disables the context manager’s functionality and should only be set when the user knows the inputs will not be uneven (default: True).

- throw_on_early_termination ([bool](https://docs.python.org/3/builtins/functions.html#bool)) – a flag controlling whether to throw an exception upon detecting uneven inputs (default: False).

```python
static notify_join_context(joinable: Joinable)
```

Notifies the join context manager that the calling process has not yet joined.

Then, if throw_on_early_termination=True, checks if uneven inputs have been detected
(i.e. if one process has already joined) and throws an exception if so.

This method should be called from a [Joinable](/docs/generated/tensorplay.distributed.algorithms.Joinable.html#tensorplay.distributed.algorithms.Joinable) object before
its per-iteration collective communications. For example, this should
be called at the beginning of the forward pass in
DistributedDataParallel.

Only the first [Joinable](/docs/generated/tensorplay.distributed.algorithms.Joinable.html#tensorplay.distributed.algorithms.Joinable) object passed into the context
manager performs the collective communications in this method, and
for the others, this method is vacuous.

Parameters:

joinable ([Joinable](/docs/generated/tensorplay.distributed.algorithms.Joinable.html#tensorplay.distributed.algorithms.Joinable)) – the [Joinable](/docs/generated/tensorplay.distributed.algorithms.Joinable.html#tensorplay.distributed.algorithms.Joinable) object calling this
method.

Returns:

An async work handle for the all-reduce meant to notify the context
manager that the process has not yet joined if joinable is the
first one passed into the context manager; None otherwise.

[#](#api-tensorplay.distributed.algorithms.Joinable)

### Joinable

class[Full reference ↗](/docs/generated/tensorplay.distributed.algorithms.Joinable.html)

```python
class tensorplay.distributed.algorithms.Joinable
```

This defines an abstract base class for joinable classes.

A joinable class
(inheriting from [Joinable](#tensorplay.distributed.algorithms.Joinable)) should implement [join_hook()](#tensorplay.distributed.algorithms.Joinable.join_hook),
which returns a [JoinHook](/docs/generated/tensorplay.distributed.algorithms.JoinHook.html#tensorplay.distributed.algorithms.JoinHook) instance, in addition to
[join_device()](#tensorplay.distributed.algorithms.Joinable.join_device) and [join_process_group()](#tensorplay.distributed.algorithms.Joinable.join_process_group) that return device and
process group information, respectively.

```python
abstract property join_device
```

Return the device from which to perform collective communications needed by the join context manager.

```python
abstractmethod join_hook(**kwargs) → JoinHook
```

Return a [JoinHook](/docs/generated/tensorplay.distributed.algorithms.JoinHook.html#tensorplay.distributed.algorithms.JoinHook) instance for the given [Joinable](#tensorplay.distributed.algorithms.Joinable).

Parameters:

kwargs ([dict](https://docs.python.org/3/builtins/stdtypes.html#dict)) – a [dict](https://docs.python.org/3/builtins/stdtypes.html#dict) containing any keyword arguments
to modify the behavior of the join hook at run time; all
[Joinable](#tensorplay.distributed.algorithms.Joinable) instances sharing the same join context
manager are forwarded the same value for kwargs.

```python
abstract property join_process_group: Any
```

Returns the process group for the collective communications needed by the join context manager itself.

[#](#api-tensorplay.distributed.algorithms.JoinHook)

### JoinHook

class[Full reference ↗](/docs/generated/tensorplay.distributed.algorithms.JoinHook.html)

```python
class tensorplay.distributed.algorithms.JoinHook
```

This defines a join hook, which provides two entry points in the join context manager.

Entry points : a main hook, which is called repeatedly while there exists a non-joined
process, and a post-hook, which is called once all processes have joined.

To implement a join hook for the generic join context manager, define a
class that inherits from [JoinHook](#tensorplay.distributed.algorithms.JoinHook) and override main_hook() and
post_hook() as appropriate.

```python
main_hook() → None
```

Call this hook while there exists a non-joined process to shadow collective communications in a training iteration.

Training iteration i.e., in one forward pass, backward pass, and optimizer step.

```python
post_hook(is_last_joiner: bool) → None
```

Call hook after all processes have joined.

It is passed an additional bool argument is_last_joiner, which indicates if the rank is one of the last to join.

Parameters:

is_last_joiner ([bool](https://docs.python.org/3/builtins/functions.html#bool)) – True if the rank is one of the last to
join; False otherwise.