Website Blog TensorPlay documentation

Automatic differentiation package - tensorplay.autograd Automatic Mixed Precision package - tensorplay.amp Distributed communication package - tensorplay.distributed Multiprocessing package - tensorplay.multiprocessing Quantization tensorplay tensorplay.audio tensorplay.compiler tensorplay.cuda tensorplay.fft tensorplay.futures tensorplay.hub tensorplay.library tensorplay.linalg tensorplay.nn tensorplay.nn.functional tensorplay.nn.init tensorplay.optim tensorplay.random tensorplay.sparse tensorplay.special tensorplay.stax tensorplay.utils.checkpoint tensorplay.utils.data tensorplay.vision Upstream anchor labels

Reference guides

Copy

View Markdown Download .md

tensorplay.optim

Base class

tensorplay.optim.optimizer.Optimizer

Base class for optimizers.

Module-level hooks

`tensorplay.optim.optimizer.register_optimizer_step_post_hook`
`tensorplay.optim.optimizer.register_optimizer_step_pre_hook`

Utilities

tensorplay.optim.swap_in_optimizer_params_and_state

Temporarily install replacement parameters and packed optimizer state.

Algorithms

`tensorplay.optim.adadelta.Adadelta`	Adadelta optimizer matching `torch.optim.Adadelta`.
`tensorplay.optim.Adafactor`	Implements Adafactor algorithm.
`tensorplay.optim.adagrad.Adagrad`
`tensorplay.optim.adam.Adam`
`tensorplay.optim.adamw.AdamW`	Adam with decoupled weight decay, matching `torch.optim.AdamW`.
`tensorplay.optim.sparse_adam.SparseAdam`	SparseAdam implements a masked version of the Adam algorithm suitable for sparse gradients.
`tensorplay.optim.adamax.Adamax`	Adamax optimizer matching `torch.optim.Adamax`.
`tensorplay.optim.asgd.ASGD`	Averaged stochastic gradient descent.
`tensorplay.optim.lbfgs.LBFGS`	Limited-memory BFGS optimizer, aligned with `torch.optim.LBFGS`.
`tensorplay.optim.Muon`	Implements Muon algorithm.
`tensorplay.optim.nadam.NAdam`
`tensorplay.optim.radam.RAdam`
`tensorplay.optim.rmsprop.RMSprop`	RMSprop optimizer matching Torch's centered and momentum variants.
`tensorplay.optim.rprop.Rprop`	Resilient backpropagation optimizer.
`tensorplay.optim.sgd.SGD`

How to adjust learning rate

`tensorplay.optim.lr_scheduler.LRScheduler`	Base class for all learning rate schedulers.
`tensorplay.optim.lr_scheduler.LambdaLR`	Sets the initial learning rate.
`tensorplay.optim.lr_scheduler.MultiplicativeLR`	Multiply the learning rate of each parameter group by the factor given in the specified function.
`tensorplay.optim.lr_scheduler.StepLR`	Decays the learning rate of each parameter group by gamma every step_size epochs.
`tensorplay.optim.lr_scheduler.MultiStepLR`	Decays the learning rate of each parameter group by gamma once the number of epoch reaches one of the milestones.
`tensorplay.optim.lr_scheduler.ConstantLR`	Multiply the learning rate of each parameter group by a small constant factor.
`tensorplay.optim.lr_scheduler.LinearLR`	Decays the learning rate of each parameter group by linearly changing small multiplicative factor.
`tensorplay.optim.lr_scheduler.ExponentialLR`	Decays the learning rate of each parameter group by gamma every epoch.
`tensorplay.optim.lr_scheduler.PolynomialLR`	Decays the learning rate of each parameter group using a polynomial function in the given total_iters.
`tensorplay.optim.lr_scheduler.CosineAnnealingLR`	Set the learning rate of each parameter group using a cosine annealing schedule.
`tensorplay.optim.lr_scheduler.ChainedScheduler`	Chains a list of learning rate schedulers.
`tensorplay.optim.lr_scheduler.SequentialLR`	Contains a list of schedulers expected to be called sequentially during the optimization process.
`tensorplay.optim.lr_scheduler.ReduceLROnPlateau`	Reduce learning rate when a metric has stopped improving.
`tensorplay.optim.lr_scheduler.CyclicLR`	Sets the learning rate of each parameter group according to cyclical learning rate policy (CLR).
`tensorplay.optim.lr_scheduler.OneCycleLR`	Sets the learning rate of each parameter group according to the 1cycle learning rate policy.
`tensorplay.optim.lr_scheduler.CosineAnnealingWarmRestarts`	Set the learning rate of each parameter group using a cosine annealing schedule.

Putting it all together: EMA

`tensorplay.optim.swa_utils.AveragedModel`	A model that maintains a running SWA or EMA copy of another model.
`tensorplay.optim.swa_utils.SWALR`	Anneal each optimizer learning rate to a fixed SWA learning rate.
`tensorplay.optim.swa_utils.get_ema_avg_fn`
`tensorplay.optim.swa_utils.get_swa_avg_fn`
`tensorplay.optim.swa_utils.get_swa_multi_avg_fn`
`tensorplay.optim.swa_utils.get_ema_multi_avg_fn`
`tensorplay.optim.swa_utils.update_bn`

Help improve this page

Found an error, an unclear step, or a missing example?

Was this page helpful?

Suggest an edit Report an issue

tensorplay.nn.init

tensorplay.nn.init.calculate_gain

tensorplay.random

tensorplay.manual_seed