# tensorplay.optim

Source: https://www.tensorplay.cn/docs/optim.html

# tensorplay.optim

## Base class

[tensorplay.optim.optimizer.Optimizer](generated/tensorplay.optim.optimizer.Optimizer.html#tensorplay.optim.optimizer.Optimizer)

Base class for optimizers.

## Module-level hooks

[tensorplay.optim.optimizer.register_optimizer_step_post_hook](generated/tensorplay.optim.optimizer.register_optimizer_step_post_hook.html#tensorplay.optim.optimizer.register_optimizer_step_post_hook)

[tensorplay.optim.optimizer.register_optimizer_step_pre_hook](generated/tensorplay.optim.optimizer.register_optimizer_step_pre_hook.html#tensorplay.optim.optimizer.register_optimizer_step_pre_hook)

## Utilities

[tensorplay.optim.swap_in_optimizer_params_and_state](generated/tensorplay.optim.swap_in_optimizer_params_and_state.html#tensorplay.optim.swap_in_optimizer_params_and_state)

Temporarily install replacement parameters and packed optimizer state.

## Algorithms

[tensorplay.optim.adadelta.Adadelta](generated/tensorplay.optim.adadelta.Adadelta.html#tensorplay.optim.adadelta.Adadelta)

Adadelta optimizer matching torch.optim.Adadelta.

[tensorplay.optim.Adafactor](generated/tensorplay.optim.Adafactor.html#tensorplay.optim.Adafactor)

Implements Adafactor algorithm.

[tensorplay.optim.adagrad.Adagrad](generated/tensorplay.optim.adagrad.Adagrad.html#tensorplay.optim.adagrad.Adagrad)

[tensorplay.optim.adam.Adam](generated/tensorplay.optim.adam.Adam.html#tensorplay.optim.adam.Adam)

[tensorplay.optim.adamw.AdamW](generated/tensorplay.optim.adamw.AdamW.html#tensorplay.optim.adamw.AdamW)

Adam with decoupled weight decay, matching torch.optim.AdamW.

[tensorplay.optim.sparse_adam.SparseAdam](generated/tensorplay.optim.sparse_adam.SparseAdam.html#tensorplay.optim.sparse_adam.SparseAdam)

SparseAdam implements a masked version of the Adam algorithm suitable for sparse gradients.

[tensorplay.optim.adamax.Adamax](generated/tensorplay.optim.adamax.Adamax.html#tensorplay.optim.adamax.Adamax)

Adamax optimizer matching torch.optim.Adamax.

[tensorplay.optim.asgd.ASGD](generated/tensorplay.optim.asgd.ASGD.html#tensorplay.optim.asgd.ASGD)

Averaged stochastic gradient descent.

[tensorplay.optim.lbfgs.LBFGS](generated/tensorplay.optim.lbfgs.LBFGS.html#tensorplay.optim.lbfgs.LBFGS)

Limited-memory BFGS optimizer, aligned with torch.optim.LBFGS.

[tensorplay.optim.Muon](generated/tensorplay.optim.Muon.html#tensorplay.optim.Muon)

Implements Muon algorithm.

[tensorplay.optim.nadam.NAdam](generated/tensorplay.optim.nadam.NAdam.html#tensorplay.optim.nadam.NAdam)

[tensorplay.optim.radam.RAdam](generated/tensorplay.optim.radam.RAdam.html#tensorplay.optim.radam.RAdam)

[tensorplay.optim.rmsprop.RMSprop](generated/tensorplay.optim.rmsprop.RMSprop.html#tensorplay.optim.rmsprop.RMSprop)

RMSprop optimizer matching Torch's centered and momentum variants.

[tensorplay.optim.rprop.Rprop](generated/tensorplay.optim.rprop.Rprop.html#tensorplay.optim.rprop.Rprop)

Resilient backpropagation optimizer.

[tensorplay.optim.sgd.SGD](generated/tensorplay.optim.sgd.SGD.html#tensorplay.optim.sgd.SGD)

## How to adjust learning rate

[tensorplay.optim.lr_scheduler.LRScheduler](generated/tensorplay.optim.lr_scheduler.LRScheduler.html#tensorplay.optim.lr_scheduler.LRScheduler)

Base class for all learning rate schedulers.

[tensorplay.optim.lr_scheduler.LambdaLR](generated/tensorplay.optim.lr_scheduler.LambdaLR.html#tensorplay.optim.lr_scheduler.LambdaLR)

Sets the initial learning rate.

[tensorplay.optim.lr_scheduler.MultiplicativeLR](generated/tensorplay.optim.lr_scheduler.MultiplicativeLR.html#tensorplay.optim.lr_scheduler.MultiplicativeLR)

Multiply the learning rate of each parameter group by the factor given in the specified function.

[tensorplay.optim.lr_scheduler.StepLR](generated/tensorplay.optim.lr_scheduler.StepLR.html#tensorplay.optim.lr_scheduler.StepLR)

Decays the learning rate of each parameter group by gamma every step_size epochs.

[tensorplay.optim.lr_scheduler.MultiStepLR](generated/tensorplay.optim.lr_scheduler.MultiStepLR.html#tensorplay.optim.lr_scheduler.MultiStepLR)

Decays the learning rate of each parameter group by gamma once the number of epoch reaches one of the milestones.

[tensorplay.optim.lr_scheduler.ConstantLR](generated/tensorplay.optim.lr_scheduler.ConstantLR.html#tensorplay.optim.lr_scheduler.ConstantLR)

Multiply the learning rate of each parameter group by a small constant factor.

[tensorplay.optim.lr_scheduler.LinearLR](generated/tensorplay.optim.lr_scheduler.LinearLR.html#tensorplay.optim.lr_scheduler.LinearLR)

Decays the learning rate of each parameter group by linearly changing small multiplicative factor.

[tensorplay.optim.lr_scheduler.ExponentialLR](generated/tensorplay.optim.lr_scheduler.ExponentialLR.html#tensorplay.optim.lr_scheduler.ExponentialLR)

Decays the learning rate of each parameter group by gamma every epoch.

[tensorplay.optim.lr_scheduler.PolynomialLR](generated/tensorplay.optim.lr_scheduler.PolynomialLR.html#tensorplay.optim.lr_scheduler.PolynomialLR)

Decays the learning rate of each parameter group using a polynomial function in the given total_iters.

[tensorplay.optim.lr_scheduler.CosineAnnealingLR](generated/tensorplay.optim.lr_scheduler.CosineAnnealingLR.html#tensorplay.optim.lr_scheduler.CosineAnnealingLR)

Set the learning rate of each parameter group using a cosine annealing schedule.

[tensorplay.optim.lr_scheduler.ChainedScheduler](generated/tensorplay.optim.lr_scheduler.ChainedScheduler.html#tensorplay.optim.lr_scheduler.ChainedScheduler)

Chains a list of learning rate schedulers.

[tensorplay.optim.lr_scheduler.SequentialLR](generated/tensorplay.optim.lr_scheduler.SequentialLR.html#tensorplay.optim.lr_scheduler.SequentialLR)

Contains a list of schedulers expected to be called sequentially during the optimization process.

[tensorplay.optim.lr_scheduler.ReduceLROnPlateau](generated/tensorplay.optim.lr_scheduler.ReduceLROnPlateau.html#tensorplay.optim.lr_scheduler.ReduceLROnPlateau)

Reduce learning rate when a metric has stopped improving.

[tensorplay.optim.lr_scheduler.CyclicLR](generated/tensorplay.optim.lr_scheduler.CyclicLR.html#tensorplay.optim.lr_scheduler.CyclicLR)

Sets the learning rate of each parameter group according to cyclical learning rate policy (CLR).

[tensorplay.optim.lr_scheduler.OneCycleLR](generated/tensorplay.optim.lr_scheduler.OneCycleLR.html#tensorplay.optim.lr_scheduler.OneCycleLR)

Sets the learning rate of each parameter group according to the 1cycle learning rate policy.

[tensorplay.optim.lr_scheduler.CosineAnnealingWarmRestarts](generated/tensorplay.optim.lr_scheduler.CosineAnnealingWarmRestarts.html#tensorplay.optim.lr_scheduler.CosineAnnealingWarmRestarts)

Set the learning rate of each parameter group using a cosine annealing schedule.

### Putting it all together: EMA

[tensorplay.optim.swa_utils.AveragedModel](generated/tensorplay.optim.swa_utils.AveragedModel.html#tensorplay.optim.swa_utils.AveragedModel)

A model that maintains a running SWA or EMA copy of another model.

[tensorplay.optim.swa_utils.SWALR](generated/tensorplay.optim.swa_utils.SWALR.html#tensorplay.optim.swa_utils.SWALR)

Anneal each optimizer learning rate to a fixed SWA learning rate.

[tensorplay.optim.swa_utils.get_ema_avg_fn](generated/tensorplay.optim.swa_utils.get_ema_avg_fn.html#tensorplay.optim.swa_utils.get_ema_avg_fn)

[tensorplay.optim.swa_utils.get_swa_avg_fn](generated/tensorplay.optim.swa_utils.get_swa_avg_fn.html#tensorplay.optim.swa_utils.get_swa_avg_fn)

[tensorplay.optim.swa_utils.get_swa_multi_avg_fn](generated/tensorplay.optim.swa_utils.get_swa_multi_avg_fn.html#tensorplay.optim.swa_utils.get_swa_multi_avg_fn)

[tensorplay.optim.swa_utils.get_ema_multi_avg_fn](generated/tensorplay.optim.swa_utils.get_ema_multi_avg_fn.html#tensorplay.optim.swa_utils.get_ema_multi_avg_fn)

[tensorplay.optim.swa_utils.update_bn](generated/tensorplay.optim.swa_utils.update_bn.html#tensorplay.optim.swa_utils.update_bn)