PostLocalSGDState

class tensorplay.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook.PostLocalSGDState(process_group, subgroup, start_localSGD_iter, post_local_gradient_allreduce=True)[source]

Store state for all-reducing gradients globally until given step, then locally after.

Stores the state for all-reducing gradients globally using process_group until step start_localSGD_iter, and all-reducing gradients locally using subgroup afterwards.

If process_group is None, the global process group will be used. If subgroup is None, the intra-node process group on each machine will be used.

Additionally, post_local_gradient_allreduce may be worth tuning, because both true and false may give a faster convergence.

maybe_increase_iter(bucket)[source]

Track iterations and trigger log message at start of local SGD.