API symbolsdistributed
Copy
tensorplay.distributed.algorithms.model_averaging.utils.average_parameters
- tensorplay.distributed.algorithms.model_averaging.utils.average_parameters(params: Iterator[Parameter], process_group: ProcessGroup)[source]
Averages all the given parameters.
For allreduce efficiency, all the parameters are flattened into a contiguous buffer. Thus, it requires extra memory of the same size as the given parameters.
Help improve this page
Found an error, an unclear step, or a missing example?
Was this page helpful?
TCPStore
Client for a threaded TCP key-value server ( torch.distributed.TCPStore subset: set/get/add/delete/check as used by rendezvous and barriers).
tensorplay.distributed.algorithms.model_averaging.utils.average_parameters_or_parameter_groups
Averages parameters of a model or parameter groups of an optimizer.